Rpmbuild large packages

According to this discussion, the limit would be 4GiB, even though sizes are represented in 64 bits. But RPM is based on cpio(5), and that format is limited to 4GiB. The release notes for rpm-4.6.0 claim "larger sizes than 2GiB", and "larger files".

In any case, I'd suggest splitting this monster into manageable pieces. Say along user/development/runtime or so. If you are including libraries, specially third party libraries, Just Don't Do That. I remember a discussion on LWN on this a while back, but can't find it now. Short version: Use the system's libraries/packages, they will be updated promptly, your own version won't. Use the work of others, don't redo the wheel.

vonbrand

Posted 2013-02-22T21:02:38.710

Reputation: 2 083

I managed to bring down the size of the package to 3.1G and am splitting whatever large files there are to smaller 512MB files [with split]. Still see the bad magic error. The issue is a single object dump file is causing the package to be large. .I am using system libraries for most part. [and numpy]. :) Sorry if I sound naive, but i am new to packaging. Although I think I might see a few alternatives I might try out to...thanks for the suggestion. Also as I understand cpio allows us to specify archive format with -H, is/should there a way to tell rpmbuild to specify archiving format? – Ark – 2013-02-23T02:36:09.017

[By size of the package I mean the total size of all the files that go in the package] – Ark – 2013-02-23T02:38:47.080

No, rpm uses the cpio(5) format. What "whatever large files" do ypu have that give you this huge RPM? Can't you just do a set of packages, and make a collection package that requires all of them? Or have them interdependent in some other way to get all what is needed installed? – vonbrand – 2013-02-23T02:43:30.207

Oh..I see...and by large file, I meant a numpy.ndarray which is generated as a result of classification [stochastic gradient]. I guess I can do set of packages. This is only the initial run, of course later on I will try to bring down the size or if that is not possible see better alternatives.Thanks, I think this would work. – Ark – 2013-02-23T02:48:45.067

Does it make sense to package generated files? Maybe tell the user that "you can find example files at so and so website"? If it does, is there some structure in the file that can be used to represent it compactly? Ship it compressed, perhaps after sorting or such? – vonbrand – 2013-02-23T02:56:40.387

It does make sense in my case to package generated files, I serialized the structure and dumped only essential parts, so that I can re-create the object later on. I used compression external as well as on object, did not get a significant change. Am trying to limit the amount of data it holds without sacrificing accuracy, which is test-in-progress. – Ark – 2013-02-23T03:02:00.973

[ Due to resource limitations, I cannot host it as a service :( ] – Ark – 2013-02-23T03:07:12.947

Rpmbuild large packages

Answers