Operating Systems => Linux and UNIX => Topic started by: piratePenguin on 17 November 2005, 21:52
Title: Source-package management
Post by: piratePenguin on 17 November 2005, 21:52
I've had a few ideas (and I'm working on them). Posted them to my ancient blog @ blogger.com: http://piratepenguin.blogspot.com/2005/11/source-package-management.html (http://piratepenguin.blogspot.com/2005/11/source-package-management.html)
soooooooooo...... thoughts? Think it'd work well, if it was implemented?
Title: Re: Source-package management
Post by: piratePenguin on 18 November 2005, 01:57
OK, here's a better XML file if anyone's interested in this thing, for irssi.
advice="recommended"> Support for perl extensions for irssi.
Just use your imagination for now to find out what exactly the client can do with this information.
Quote from: worker201
Perhaps something like this could work. Here's what you would need to do, I think. Have your distribution system work like php. Then the source code to all these programs gets dropped into the database. When my computer running FC4 with stops by to pick up the latest release of transcode, the package manager looks at my system, and determines what flags are required to create a package custom-suited to my needs. The package manager then gives these requirements to the distro system, which produces a package custom-fit for me. It would also store a compressed copy of the package in the database, just in case someone else with similar requirements comes for the package.
source (http://www.microsuck.com/forums/showpost.php?p=98622&postcount=22) Well worker, with this (the thing I was describing back then was nothing like this) system, such a system wouldn't be hard to do.
The hardest bit, by far, would be the configure bit of the XML files. For big packages with many options, that'd be one heck of a bitch.
Oh man. Gaping opportunity for an XUL(or HTML and friends, but that's like, old technology :p. The XUL app could easily be run remotely (what worker described) or installed locally (far easier than with HTML/PHP/whatever), and it'd look like a native application.)-built frontend to this thing.
Title: Re: Source-package management
Post by: worker201 on 18 November 2005, 02:37
It looks like what you are doing here is making the package do the work. Meaning that the configuration files are package dependent. Then your distroserver has to parse the files and then get down to business. What you are basically doing is asking developers to do it your way, and not their way. They don't like that. If they could agree to a certain way of doing things, they probably would have already.
I think there is a better way. It should be possible for the distroserver to run a ./configure just like any monkey, and then generate the configuration files by itself. Once a config file was created, it could be stored, to make installation of that package faster for the next guy.
I think this would maybe make the whole system more efficient. Because just about every program built with autoconf has --build-target=, --with-shared, --prefix=, and etc. The system can prepare canned answers to each of these, based on the machine, so you get a custom package. That leaves extra time and trouble for the optional components.
So, instead of going to the distroserver and asking for an FC3 package, the distro asks your computer what kind of package it wants. Then it builds a custom rpm (as an example) that is designed for your computer, on the fly, while you wait. This is time-consuming, but you can always use a little bandwidth to entertain during the process.
The reason to do it this way is that all you have to do are find the source tarballs. Every open developer has them. It wouldn't require anything more from anybody else than the current system (or lack thereof) already has. All the burden is placed on you and your program. Yeah, it will be harder for you. But it will be completely unobtrusive to everyone else. I mean, imagine what a pain in the ass it would be, if that sorta lazy programmer at ffmpeg didn't write an XML config file. You'd have to write one for him, and since he doesn't release anything but CVS snapshots, that would maybe be once a week. That's too much work, all because some lazy guy won't do things your way. It would be so much easier to just grab the CVS tarball, and have the machine do all the work.
All the users would have to do is download the app and/or visit the website - everything else is taken care of. It's like the way apt repos are done now, except there's no reason to keep extra binaries on hand for all the different possible flavors of Linux.
And best of all, if there is an important option for the package that I want, I get it. Example there is dvdauthor. It requires png15-devel, but only if the --enable-png flag is set at build time. Dag and Dries, in Fedora aptland, just didn't enable png, eliminating the devel dependency. But then you can't use pngs in your dvd menus. I had to download dvdauthor myself and configure it - the apt package was useless to me.
Just a few thoughts and experiences I have had that might help you design your system.
Title: Re: Source-package management
Post by: piratePenguin on 18 November 2005, 03:08
Quote from: worker201
the configuration files are package dependent.
Yea, just like the FreeBSD port's makefiles. Although these XML package files will be far more complicated. But whoever's writing the XML file doesn't have to cover every configure flag. Whatever they think is necessary, just. The frontend could produce the output of './configure --help', and the user could add their own options.
Quote from: worker201
What you are basically doing is asking developers to do it your way, and not their way.
I'm not asking anyone to do anything.
This isn't a bid by piratePenguin to take over the world. It's not 2010 yet.
Quote from: worker201
I think there is a better way. It should be possible for the distroserver to run a ./configure just like any monkey, and then generate the configuration files by itself. Once a config file was created, it could be stored, to make installation of that package faster for the next guy.
I dunno what different system you're talking about.
The XML files I posted are used by the package manager to sort out dependencies and configure options. It asks the user do they want to apply some_patch, do they want to use '--enable-png' and install libpng. Then it patches, installs dependencies (libpng), runs configure with the options the user wanted, and then compiles it.
Quote from: worker201
So, instead of going to the distroserver and asking for an FC3 package, the distro asks your computer what kind of package it wants. Then it builds a custom rpm (as an example) that is designed for your computer, on the fly, while you wait. This is time-consuming, but you can always use a little bandwidth to entertain during the process.
All that was part of the plan.
Quote from: worker201
imagine what a pain in the ass it would be, if that sorta lazy programmer at ffmpeg didn't write an XML config file.
I don't expect him to. I could probably set it up so old ffmpeg package files (that's the XML file) will work with new ones (although they won't know about e.g. --enable-something-new until the next update of the package file).
Quote from: worker201
And best of all, if there is an important option for the package that I want, I get it. Example there is dvdauthor. It requires png15-devel, but only if the --enable-png flag is set at build time.
Now any frontend can and should (unless you configured the package manager (not the frontend) to automatically accept/reject recommended configure flags) ask you about '--enable-png' and you can decide if you want it or not. If you say yes, it'll check if you have libpng. If you don't have it, it'll install it. If you have an old version, it'll update it.
Quote from: worker201
Dag and Dries, in Fedora aptland, just didn't enable png, eliminating the devel dependency. But then you can't use pngs in your dvd menus. I had to download dvdauthor myself and configure it - the apt package was useless to me.
Well in this source package manager (remember that's what it will be), you won't have to worry about that.
Quote from: worker201
Just a few thoughts and experiences I have had that might help you design your system.
Thanks for your input. A few of the bits I didn't understand (most of the bits I didn't quote/comment on here) - but I think that might be because you misinterpreted what I'm trying to do...
Title: Re: Source-package management
Post by: worker201 on 18 November 2005, 21:55
You're going to write the xml files for each package? That's a shitload of work. I thought your plan was to have the devs write the xml files. But if you want to do the work yourself, then that's cool, I guess.
Title: Re: Source-package management
Post by: piratePenguin on 18 November 2005, 22:01
Quote from: worker201
You're going to write the xml files for each package? That's a shitload of work. I thought your plan was to have the devs write the xml files. But if you want to do the work yourself, then that's cool, I guess.
Yea I was always gonna write the XML files. Only question is how much info (configure stuff mainly) I shoulld put in them...
After I have XML files for the (relatively) few packages required for a usable system (that doesn't mean graphical stuff... But Xorg and some WMs, I wouldn''t mind doing) (and the package manager written, of course) I could make a distro that uses them.
Title: Re: Source-package management
Post by: worker201 on 18 November 2005, 23:08
It would probably be easier in the long run to create a script that dynamically creates the xml file. All the appropriate information could be pulled straight from the configure file, given the right kind of reader. Then all you have to do is feed a full tarball to the script, and it would build the package for you.
Title: Re: Source-package management
Post by: piratePenguin on 20 November 2005, 01:21
Quote from: worker201
It would probably be easier in the long run to create a script that dynamically creates the xml file. All the appropriate information could be pulled straight from the configure file, given the right kind of reader. Then all you have to do is feed a full tarball to the script, and it would build the package for you.
Good idea.
I think I know what way to do this now. 2 programs (actually they'll be libraries in the end, and then there will be a frontend) - one for the configuration, and one for everything else (solving dependencies, package management itself). Whenever the configuration program doesn't have an XML file, it'll notify the user and print the output of './configure --help', and the user can decide what configuration options to use. Or maybe pull the info like you said, and give that to the user.
Title: Re: Source-package management
Post by: piratePenguin on 24 November 2005, 22:19
The idea so far:
The frontend is what the user sees and interacts with. The frontend interacts with two libraries - libdependencies and libconfigure (which I'm not so sure about).
libdependencies is used to sort out dependencies. It will be extensible - there will be extensions for it to support packages, users, and various other stuff. libdependencies asks the extensions "do you have X?" (X being a dependency - the name of a package or the location of a file, or whatever the plugin takes.), and the extension replies as appropriate.
If the extension replies with "no", then libdependencies checks it's configuration to see whether it should tell the extension to go fulfill the dependency or ask the user first. All dependencies that get to libdependencies are required (hence libconfigure (part of it's reason for existance is to act as a frontend to libdependencies), which just might be replaced with libsourcepackagemanagement, or something), so when libdependencies tells the frontend that a dependency isn't met right now, the user will only have a choice between having the dependency solved or cancelling installation (or, if it's a configure option that triggered the dependency, then just cancel the configure option).
All the extensions to libdependencies will actually be libraries (I think that's how extensability works most of the time). The extension that deals with packages, say it's called libdeppackages, will itself be extensible. You will be able to install an extension for RPM, which would check the RPM database for packages, Slackware, which would check /var/log/packages for packages, etc. That way, when some package asks for glib2, and you have it installed as an RPM, this package manager won't go get glib2 again, or anything. Another way that could be dealt with is searching for files a package installs, but that'd be too messy.
Title: Re: Source-package management
Post by: Jack2000 on 24 November 2005, 22:41
nice work pP
Title: Re: Source-package management
Post by: piratePenguin on 24 November 2005, 23:08
Quote from: Jack2000
nice work pP
I haven't actually done anything yet. Just thinking. And now I wanna implement this - but I needa learn quite a bit more programming first (actually I don't, and I probably won't (but I will be looking at code from XMMS and other random programs that are extensible), which means it's gonna be messy at first. I have no experience with this stuff.).
And I just might abandon it half way too, but hopefully not :)
Title: Re: Source-package management
Post by: Master of Reality on 4 December 2005, 06:00
Sounds a bit like the Gentoo portage system ;)
Title: Re: Source-package management
Post by: piratePenguin on 4 December 2005, 16:35
Quote from: Master of Reality
Sounds a bit like the Gentoo portage system ;)
Yep it would be, a bit. I've never used Gentoo, but I have used FreeBSD. What's the difference between their port collection stuff?
In my package management system, you'd just ask the frontend to get you e.g. irssi, and it'll go get it. A server would be contacted and the XML files downloaded and parsed, dependencies solved, configure flags sorted out (usually off the user), anything else that needs to be done, and finally it'll be compiled and installed. The XML files won't (normally) be stored on the local filesystem like they are in FreeBSD and I think Gentoo, they'd be on a server and downloaded as required.
Title: Re: Source-package management
Post by: Master of Reality on 4 December 2005, 16:39
I've never really used Gentoo either, but i think im switching to it soon. Its package system is inspired by FreeBSD and the ports system. So its fairly close. According to the gentoo site the only thing it is lacking at all in is with binary packages. They are starting to support more, but it prabably does not have as much support as the BSD package system yet.
Title: Re: Source-package management
Post by: piratePenguin on 4 December 2005, 16:44
Quote from: Master of Reality
According to the gentoo site the only thing it is lacking at all in is with binary packages.
I've had an idea for binary packages, but I haven't thought about it much. Any configure flags used, and patches applied, and packages required etc., would be stored in a file and saved in a tarball with the binaries/etc. A frontend (which could be a web-based interface) could poke into the tarballs and give the information to the user, and sort out the runtime dependences needed.
Title: Re: Source-package management
Post by: piratePenguin on 14 October 2006, 01:51
Conary (http://wiki.conary.com/wiki/Conary) is the best package management ever. Ever. I dunno where I was going with this crap, but it can't possibly be anywhere conary hasn't covered with perfection.
The way conary works is (in a very small nutshell), the repository stores a 'recipe' file which describes how to build the package. A recipe file for GNU hello (http://www.gnu.org/software/hello/) might be (obviously this is about as simple as they come, but more times than not recipe files aren't much bigger than this):
class hello (AutoPackageRecipe): name = 'hello' version = '2.1.1'
buildRequires = ['perl:runtime']
def unpack (r): r.addArchive ('http://ftp.gnu.org/gnu/%(name)s/%(name)s-%(version)s.tar.gz')Anybody can 'cook' the recipe (compile it, basically) on their own machine from this (it's as easy as 'conary emerge hello:source'), and if a developer cooks it on the repository then users can install the binaries.
Updates come in 'changesets', which is the difference between the versions. So after just a few months you shouldn't be asked to install 100Mb+ of updates (like I believe Ubuntu 6.06 now does). Changesets can be rolled back - back to day one if you like. Parallel installation of certain packages (like stable gimp and development gimp) is a sinch. Packages can be installed by users into their home directory..
I've yet to come across anything that's really hard or impossible with it (patching, use flags, multi-arch stuff - it's all there).
But the really cool stuff comes in when you're ... putting together an OS. You can pretty much do it from a web based interface! (picking between packages that've already been built) And, when you must customise a package, that's not hard.
For all the good it'll do ya, my repository is @ http://piratepenguin.is-a-geek.com/conary (if you get internal server errors - refresh) To see the recipe files click on the source sub-menu of the package, go to show files and click the recipe file. Right now, there's a lot of shit and not much useful stuff, but some example recipes. And there's a hell of a lot of recipe files available @ http://www.rpath.com/rbuilder/
So conary covers source and binary package management. And I don't think there's any better way to cover them (there are other, more complex, user-end advantages that I haven't discussed (branch affinity (http://wiki.rpath.com/wiki/Glossary#B) for one)).
Title: Re: Source-package management
Post by: H_TeXMeX_H on 14 October 2006, 02:40
Alright ... but is it GPL ?
Title: Re: Source-package management
Post by: piratePenguin on 14 October 2006, 15:36
Quote from: H_TeXMeX_H
Alright ... but is it GPL ?
No, but it's free software and incompatible with the GPL. Common Public License 1.0 http://www.gnu.org/licenses/license-list.html