One of the great things of Linux as an ecosystem is that there is so much choice. Yet this is also its greatest weakness, as unlike on MacOS and Windows this means that simple questions often do not have a simple answer. The topic of distributing software is one of these vexing topics that never really goes away, with [Noam Lewis], as author of the Fresh editor, recently expressing their frustrations on this topic.
This is a familiar topic that we have covered before, mostly in the context of cross-platform installers. Whereas on MacOS and Windows you just pick any of a handful of standard installer frameworks that do roughly the same thing, on Linux you are dealing with a dizzying number of package managers, including distro-specific ones, NPM, Homebrew, Rust crates, third party repositories like Arch Linux’s AUR or stray APT repositories for Debian, and of course AppImage and Flatpak along with a host of other options.
After going through the entire list with help from the Fresh community, [Noam] ended up creating packages for all of them, and has a lot of thoughts. Since Fresh is a TUI-based editor, something like Flatpack is not a really good fit, while AppImage is just very slow due to mounting the image, which is annoying.
Overall, distro-specific solutions are just unpleasant in general, as they mean repeating the same work over and over, even if you can even get into a distribution channel. Ultimately the solution that [Noam] ended up at was to have a static binary with the musl runtime, along with an integrated updater, which is a solution that should be quite familiar for Windows users. Yet even if it fails some kind of purity test for certain Linux users, is it really a bad solution if it works for users of the software?
At this point I’m convinced most Linux distros intentionally go out of their way to make distributing compiled binaries as painful as possible.
Not really, if anything they go out of their way to make it easy for you to comply with their packaging requirements. The problem is numerous standards to target, some of which are more focused on security/isolation than performance etc not that the standards are hostile.
That said last I checked at least all the ‘native’ package mangers formats are not that hard to convert between each other – not 100% certain success rate for the end user. But in designing and compiling your app you should be able to avoid those pitfalls will make supporting only .deb but getting .rpm etc for a wider audience easy enough.
And while the overhead for containerised applications can end up meaningful, I rather doubt its meaningful enough to really really matter in most cases as even the budget/second hand old but modern enough to really use daily hardware has so much performance and most applications in their own right need rather little – still less wasteful of computing cycles than current ‘AI’.
Every disto is its own “borg” that tries to assimilate every software package into their version tree, which makes cross-distro distribution almost impossible unless you statically link all the libraries you need.
Linux as an operating system (not kernel) is like a shanty town built house on top of house over decades using ad-hoc solutions without planning. It exists in a 3rd world country without a functional government, run by local warlords and gangs (distros) and it’s actively hostile against anything that tries to improve it, because doing so would mean bulldozing over someone’s house before any development can happen. If you want to get anything done, you basically have to build your own walled community inside it and then pretend like the rest of it doesn’t exist: statically link everything you can, and use the lowest common approach (oldest possible version) for things you cannot.
https://jangafx.com/insights/linux-binary-compatibility
“The Atrocious State Of Binary Compatibility on Linux and How To Address It.”
I really like cpack for producing packages without too many distro/os specifics.
I think I’m using cpack as part of my cmake build, but I still haven’t figured out a nice way of having the .deb files automatically hosted so that a client machine can easily automatically be updated.
this is exactly too painfully relatable
The flip side of the hurdles posed by Linux packaging systems is superior (compared at least to Windows) security. Attempts to reduce the friction (pypi, npm) don’t end up well.
Security through obfuscation isn’t security.
I don’t inherently thing relying on the latency of a release cycle is bad security, but it’s if ineffective means nobody is looking at it.
Linux packaging systems aren’t obfuscated. They may be tedious to use, especially if you want to build your package for a dozen of distributions (check out openSUSE’s OBS for that), but with pretty good documentation and tons of examples, there is nothing obscure about them.
The real question is, why is this Noam’s problem? It’s open source, folks who want it packaged for their favorite distro can do it themselves or ask if someone else is interested in doing it. Norm isn’t obligated to package their app for every distro under the sun, they can do it just for their favorite distro, and let the community do the rest.
In saying that, the move to a static build is a great option and reduces friction. There are tools like AppMan which are designed to automatically install/manage/update binaries from git repos, so that’s a another option for users.
The proverbial Wikipedia
[Who?]
“The Community” isn’t just technically minded, ecosystem-focused participants.
This oversight is characteristic of the Linux fanboi community. Realize you’re doing yourself (and your community) a disservice by using this argument.
Case in point: you’ve ignored that any potential users might not know how to, or care to, package it. Or know anyone who would (is this a volunteering perhaps? :) ). You ignore that those acts take willful effort. It takes time, which is ever in short supply. It takes commitment, to keep the channels updated and synced (or set up the scripting to do same, or… whatever actual steps might apply). Look beyond yourself; don’t project yourself onto others. It’s not Just That Easy(TM). Different people see things differently. Give them the benefit of a doubt.
How does one grow their (app-related) community, if potential users can’t few-click load their app? They don’t. They suffer obscurity — unless the app is so killer that runaway adoption occurs spontaneously. The chance of that happening is extraordinarily low. The chance of that happening is higher, if more users can access it quickly and easily. If people are aware it even exists; advertising across…various places, as well as having it available on various repos, are contributors to success. Even if one is not targeting the “killer app” space, just quiet little utilities or, y’know simple games or whatever, there is always an effort-to-growth tradeoff. How much work do you want to put in — are you willing to put in — to grow the community?
Conversely, for participating community members, what is a desirable level of effort versus promotion/dissemination? It shouldn’t be zero; instant and broad (cross-platform? cross-community?) release would reward spammers and scammers. There should be some barrier to entry. Is this the right way to implement it? Are there other ways that might be of interest? (A fast-track for well-known participants (e.g. by suitably vetted certificate), could be an example. I don’t know the specifics myself, but I suppose that already exists in at least an ad-hoc manner for, say, kernel updates; whatever, that there is some mechanism by which some packages are more trusted or higher prioritized than others.)
This, and a drive-by-packaging effort tends to result in packages that are not updated when new versions come out. That leads to duplicate bug reports and security problems.
d3Xt3r is correct it isn’t Noam’s problem, unless they want it to be – and given the nature of what this Fresh application is the sort of people that will want it are the sort that can compile from source and/or package it themselves. So the community the program serves aught to be about as weighted to solving those problems for themselves as you can get.
And in the world of open source you don’t need to burn yourself out trying to do everything for them to then build a community – simply create something good, publish and talk about it a bit and if its actually good the other developers that have similar needs to yours…
You might well be the only real maintainer of a project that ends up pivotal to so many others, and that does seem like a stress you can live without etc. The FOSS development model does have its flaws. And yes I do not entirely disagree with you Tim you do raise decent points. But unless your time is being paid for by that community widely packaging what you made almost certainly because you had a personal need of it is entirely optional.
Is it really a hurdle if you’re just expected to know a little bit more about your distro to install something? I would argue not. And most modern distros have an “App Store” like desktop app now that hides a lot of difficult parts and homogenizes the experience. Does it mean you can’t just go to a random website and blindly download an installer for something? Yes, in that case you will need to understand what you are doing, which I would argue isn’t a bad thing. On the other hand, most apps that anybody will want to use on their desktop will likely be available from already-configured standard repos and easily installable with local tools. Linux is not Windows is not MacOS. I don’t see why it’s so difficult to understand that, nor the utility in complaining about it.
This isn’t about users installing stuff on their computers, it’s about developers packaging stuff so that users are able to install it. Of course if you use Johnathon’s Own Version Of Linux you should understand the installation process, but the person who just wrote a version of Paint for linux should not have to know the intricacies of JOVL and every other distro.
So maybe have a build chain for the most recent and most common distros? Maybe 3 or 4 versions to maintain. Don’t bother with the 7 people using UberNerdLinux, or somebody running a 10 year old release, let them figure it out if they want it. Opensource your code and let the market sort it out? How badly do you want to share your pet app? I get that if you’re a solo dev on a little indie project you don’t have a lot of free time for all the other stuff. But maybe let opensource work the way it’s intended and let people who want to help do so. I’ll also point out that there’s a snap package for the editor in the OP put out by somebody other than the original dev, so somebody has done this work.
Any one instance of “just know a little more” isn’t a significant hurdle. But in the aggregate, when every single thing you try to do on your computer has that little extra bit of friction, it adds up to a pretty significant hurdle. Maybe not too much of one for you or I to overcome, but enough that it’s more attractive to just keep using Windows for a lot of people.
Exactly! The user of Gimp (for example) does not need to be a Linux pro, they need to manipulate pictures. Demanding they learn your OS mechanics before they can do their thing is a great way of pushing them back to Windows or Macs.
99% of people are not versed in CS, just like you are not a Mech. Eng. You would never buy a car, elevator, fridge or any other stuff that required of you to know how to make them to just make them do whatever they are made for. Don’t ask people to learn OS internals just to use the OS or install some software: that’s why the year of the Linux desktop never came.
Damnit I have a master’s in CS and I have very limited patience for the mandatory yak shaving that comes with building on Linux. There is no excuse at all for this: Apple made OSX from Linux and it was even easier to live with than Windows. Too bad the machines are so overpriced and hostile to upgrades nowadays.
AppImage is extremely slow??? According to strace it takes his v0.4.9 x86_64 AppImage 0.34s from execve to drawing the menu bar. And if I extract the files and run the ELF executable (i.e. not the AppRun script) directly, it still takes 0.25s. Maybe he should stop writing 310kB of log files to ~/.local/state/fresh/logs during application startup.
I second that sentiment. 10ths are only relevent if you’re vying for pole position
I’ll take the complaint seriously when I can sip my coffee before startup.
wow! imo 0.25s is an eternity to start a text editor!
Obligatory XKCD: https://xkcd.com/927/
I still think this is the one thing windows got right. Libraries should be considered OS-level and a default set of them should be assumed on installation, allowing for the distribution of raw executable files with no dependency baggage. I run linux mint now, but I genuinely miss writing games for win32 where I could just send my friends a .exe file over AIM. Sure, that means anyone can send anyone else a .exe file, with the expected consequences, but who cares? Stupid people deserve viruses.
Windows makes installing programs easy, but uninstalling completely is very far from guaranteed! Fragments all over the place. Also, you can static compile and send your friends an executable on Linux just the same.
Well I haven’t been a regular user of Windows in decades, but the exact opposite of that was my experience. Every piece of software was responsible for delivering it’s own DLLs and properly checking that a newer version wasn’t already installed. A sloppy installer would overwrite them anyway and break other applications. Sometimes it would be weeks or months before said breakage was noticed.
(Sorry for accidentally “reporting” your post. Wish there was an undo button!)
The “DLL hell” problem you mention was a thing around the time of Windows 3.1. and 16-bit Windows. The problem was that the library had to be registered with Windows to be shared between programs, and only one version could be loaded at a time.
The problem largely went away with the NT branch of Windows, because it gave programs a private memory space, so multiple versions of a library could run concurrently. There’s actually a bit of clever memory mapping to allow two programs that use the same version to see copies of the same DLL that’s actually loaded just once, but if a third program uses a conflicting version it gets its own private copy of that version. The DLLs can exist in the program’s own folder, or in the Windows system folders, or both, and whichever version is needed is loaded.
The same problem in Linux is deeper, because of how the package management works: keeping multiple versions of a shared library concurrently is not exactly how the system was designed to operate, so it needs some sort of virtualization to work.
See:
https://en.wikipedia.org/wiki/Side-by-side_assembly
Grok build makes routine packaging work trivial, cross compiling, packaging, and testing the installs in QEMU is a one shot operation if you give Grok root on a VM to work within. Make sure you tell Grok to keep its “learnings.md” up to date then things get even more efficient. Maximise your token value position by askling Grok to prefer building parametric scripts, a toolkit, for tasks in the pipeline that are to be repeated, then you are just using inference for the input parameters and output analysis, this accumulation of tools progressively reduces token consumption, which is ultimately good for your bank balance, and the environment.
I don’t understand these complaints. I’ve been writing software for over 20 years and i never had an issue packaging software. Writing a spec file to target multiple distributions is relatively trivial, and there’s numerous tools available to automate these things and integrate building into ci/cd pipelines.
The only time I see things become problematic is when you’re doing awful wrong things, like static linking or releasing closed source software. And for those things, there SHOULD be friction.
Writing python is more like being a script kiddie than a real software developer. What you learned over 20 years of doing interpreted languages i learned in 2 months of doing Rust.
Isn’t Rust just C++ with training wheels?
Do you guys know each other, or was that just a wild baseless assumption?
If it’s about the static linking comment then I suppose that was phrased a little confrontationally, but you must admit the way rust/cargo currently does things is very much against the grain of typical Linux distros dynamic linked C/C++ package world
It was a wild baseless assumption on their part. :) I’ve been primarily a c++ developer. Although lately mostly Object-Pascal (fpc).
All these language-based package managers are indeed doing things against the grain of how things are typically packaged. Package management should be the responsibility of the distro, not the language. If everything uses a different package manager and does static linking, how the heck am I supposed to patch, say, libpng when it has a security vulnerability? I can’t because I don’t even have a way of knowing what’s linking against the vulnerable version. This trend is definitively the wrong direction.
I guess I mentioned static linking and some rust fan immediately felt attacked.
Which is odd, because, while static linking is common in that community, it /is/ in fact capable of dynamic linking.
I’ve been writing primarily c++, not python. I don’t know why you assumed python.
For a while, I installed only flatpaks because they “just work” and are one click to install. But lately, I’ve first been asking Google Search if the DNF (or APT) version installed in a Distrobox works as well on Fedora Silverblue as the flatpak version of the app. The reason that I do this is because flatpaks can only install on my local NVMe SSD, which only has 253 GB. But I’ve configured Silverblue to create my Distrobox containers on my 1-TB external SATA SSD. That way, I reserve the local NVMe for apps like Virt-Manager and Zoom that need the tight system integration that’s already designed into the flatpak. Getting the same integration from DNF would require layering, which I refuse to do. So Distrobox on the external drive is a perfect strategy to optimize my use of available storage.
I wonder that he did not talk about the, all Linux distribution have different libraries at different version level hell and the different kind of make/cmake/ninja/qmake tools in different version hell. Sometimes, often, I wish to live again in the 90 when we had no or less problem, And than the diffent programming language hell, also in different version. An example? Why need Qt with android Java? Why needs every version of Qt a different version? The word hell is not extreme enought to describe what happens today. It is hell^3!
That is what constantly bites me in the rear are the library/dependency hoops. Ah good I finally got that seemingly installed and the system didn’t balk. Go to fire it up and oh cool I have to update 4 other parts of my system to actually run the thing. It just stinks. At first packages were a godsend but now it is kind of a gamble. I hope they can figure out how to make things more universal again. Back to running Knoppix on my test machines I am so sick of curls and diggin up repos. I mean I get why they exist and change happens but when something can break every 4 days it gets a little old.
If Flatpak isn’t a good choice use a tool like nfpm to build all the distro packages easily.
If you create a genuinely useful application, other people will package it for you. Tag releases in git and/or release a source tarball periodically, and maintainers can pick it up and package it for their distro.
A lot of users (especially of stable distributions like Debian stable, appreciate having software curated by the maintainers and monitored by the maintainers, so e.g., don’t need to track every app for security issues yourself, just periodically update distro from official archives and done.
These complaints seem to come from former MS Windows users who think installing random applications off random websites is a good idea.
I think that the style of downloading a random script from a website is making more difficult to integrate an automated packaging system From the application webpage the preferred method is:
curl https://raw.githubusercontent.com/sinelaw/fresh/refs/heads/master/scripts/install.sh | sh
a ./configure && make && make install could have some security problem too and could execute harmful commands, but there tools to sandbox the execution.
Right, because packagers are always trustworthy and correctly package the software they did not have anything to do to develop. Nah, i’d rather install stuff directly from the developer than some random, outdated distro package management server.
I am exactly saying the whole package manager idea is dumb for 3rd party software as it is (it would be a good idea if it just installed the software directly from the developer for 3rd party software and only the distro stuff would be packaged by the distro), adds unnecessary, unsafe steps to the process of actually using software to do stuff on your computer. I am not interested in messing with the OS anymore. I want it to run programs, and i many times want the latest pretty fast, because there’s new features and bug fixes, not after someone has their time repackaging it.
Where do you think the distro packagers get the software from? The software developper’s “random” repository,
So what about those rare programs, that aren’t mainstream stuff, that few people use, but they are actually useful programs? Relying on some random person on the internets to package it for you, because packaging it for every distro and version is not feasable, if you plan on actually developing the software. And it’s not even that rare programs that do not get packaged or the packages updated in a reasonable time. So, who exactly is the boss of deciding which program is a “genuinely useful application”?
Your rant seems to come from a fanboy rather than someone thinking of actual usability.
As far as I understood, the main issue is that the rust ecosystem is currently not well packaged due to too many rust packages. I did only a bit of rust and it is not on the level of node with the isNumber libs but it is not far away. So for simple applications you have a dependency tree like 20 libs. So the rust community would need to package all of these libs.
This issue has already been resolved some years ago. Have a look at https://openbuildservice.org/ and it’s sibling https://build.opensuse.org/ .
I’m building my libraries there directly from git for a handful of distros, rpm based ones, deb based ones, etc. and it works great, you get repositories and everything. However getting there is still a lot of pain. Writing down the packaging descriptions (e.g. foo.spec, debian/*), with correct dependencies per codestream is anything but fun. And I do not think that this will get any better. Integrating software into larger chunks that work together is just inherently hard problem to solve.
“[Noam] ended up at was to have a static binary with the musl runtime”
Glibc breaks static compilation on purpose, my pet project was to fork Alpine or Gentoo-musl to generate static binaries:
http://stalinux.wikidot.com/
Just use Windows, it has the BEST forward compatibility.
Get things done, don’t waste your time on OS.
Life happens outside the screen.
That’s why I use Linux. I haven’t got time to baby sit Windows all the damn time.
The computers at work run Windows. Downtime caused by Windows is paid for and not my problem.
Downtime on my computers at home is time I could use for doing other things. That has to be kept to a minimum
Build it in Windows, validate it in Wine.
Usually (if you can’t provide the source) a .deb or .rpm package is enough. There are plenty of automatic repackers around.
ABout the source: that alone is often not enough, because thsis way you’ll end up maybe in AUR and then get forgotten after a while.
OP selected a toolchain that’s a footgun. Look at the Debian situation in the linked github. Debian packages are an architecture built around loosely coupled very large collection of very stable ABI functions, like C software dynamic linked to libc containing a malloc function that never changes its call signature and occasionally changes its implementation. Instead OP is doing the DLL-hell thing like python, node, rust does of every individual function is unstable incompatible and in a separate library with a specific version, so you have dozens/hundreds of packages that very slowly implement pseudo-static linking because no other binary on the user’s system will be able to link in the precise version numbers packaged.
Essentially OP is creating a static linked system the hard way using dynamic linking tools.
Normally I’d say use Docker which works well for CLI/TUI systems (even better for server systems), but that architecture is designed around preventing containers from accessing every file on the system, not encouraging it LOL. Even a language like Python which have VENV as a a workaround won’t work because the VENV won’t let his text editor out of the VENV…
If a dev selects a toolchain designed, architected, and culturally encourages to create a DLL-hell situation for the users, then complains its everyone elses fault that the DLL hell problem has not been solved by every system everywhere… No thanks. I want nothing to do with that. Use a different editor.
Some languages don’t have an ecosystem as difficult to deal with as Rust. If OP could translate to golang or C or C++ the distribution process would be more likely to succeed.
There are other architectural issues. OP should have architected his system to be a http server that users access over a browser. Its 1990 people don’t use local apps anymore everything runs in a browser. Oh wait its late 2020s that makes it even worse. Ship a docker container for amd64 and arm64 that renders in any modern browser as the “UI” and call it good. No shortage of “terminal in a browser tab” systems LOL.
This seems like it would be a good use case for Cosmopolitan / Actually Portable Executable: https://justine.lol/ape.html
It makes a statically compiled executable that works on Windows, Linux, and Mac! Though sadly IDK if there is any way to use Rust with it…
So the general consensus seems to be that Linux a) doesn’t need users who aren’t also developers or programmers themselves, and b) doesn’t need commercial for profit software, so the problem of distributing binaries does not exist.
Everybody should just compile from source, or wait until someone else does and wraps it in a package that fits your distro version. Until then, stop complaining.
Because this is apparently a hill I’m willing to die on, I’ll defend my case here. I think you’re missing the point. My argument is that if you’re a developer and think you have a killer app that everybody will want to use, then you need to either put in the work to distribute it easily and meet your users where they are, or have a standard build chain that if somebody else wants to do that work for you, it is easy to for them to do so. Linux should be easy enough for everyone to use if they want to (it’s clearly not there yet). Developers of apps that run on Linux should absolutely be able to make money doing so. These things are not mutually-exclusive.
“Back in my day…” … if the app you wanted wasn’t available in your standard distro repos, the de facto standard process to get it was fairly straightforward:
1) download a tarball
2) extract the tarball
3) ./configure
4) install missing necessary libraries and library header files, usually available from your distro’s standard repo
5) make
6) make install
Everybody knew this process and followed it. If you were a newbie, you learned it pretty quickly. The package distribution in modern distros is so much more wide now, you rarely have to do this anymore. You can probably find a 3rd party repo for your distro of choice and install the package from there. Or the developer will post a binary package somewhere for you to download and install using standard tools already on your machine. Installing stuff is generally easier now than it used to be. But you, the developer, need to put in the work.
So yes, developers should stop complaining and stop reinventing the wheel to get their application distributed. Not everybody writes apps in C/C++ anymore. There are dozens (hundreds?) of development platforms, code frameworks, languages, etc. The old way doesn’t work anymore. It’s the developer’s responsibility to make it easy for users to install their app. Plenty of tooling exists for this, you just have to learn how to use it. If your app is good enough, others will want to help you distribute it.
The repository is a bottleneck for commercial distribution, because you need someone “in the system” to maintain your packages. Either you as the software developer get yourself into a myriad of different repositories so you can do the work, or you rely on the goodwill of others to do the work for you. For small vendors with no established reputation or popularity, this is nearly impossible. Third party repositories not affiliated directly with a disto are a non-starter because they face the trouble of maintaining compatibility and staying up to date just the same – distros are hostile for outside packaging.
The fundamental system does not scale to millions of competing software titles – it can only support a grab bag of apps that are favored by the distro developer community. Everyone else has to go with the “compile it yourself” or “wrap an entire Linux OS into a container” approach.
https://news.ycombinator.com/item?id=43551934
Or alternatively, “develop for Windows, run it in Wine/Proton/Etc.” approach used by games under Linux.
https://blog.hiler.eu/win32-the-only-stable-abi/
Except WinAPI is like magnet for viruses, keyloggers and other malware commonly found in pirated games and other software.
Only Windows had the clownshow of scareware antivirus programs. Every other OS simply didn’t allow malware to run.
There are/were plenty of mac viruses.
The rest don’t suffer as much because they don’t have enough users to target with malware. It’s trivial to get a naive user to compromise a linux system because you can simply ask them to give root permissions to whatever and then the door is wide open.
” is it really a bad solution if it works for users of the software”
No, of course not. That’s the whole point!
But… as a user of software I think ‘working for me’ means playing nice with my distro’s packaging system.
Do you really have to get it into a million distro’s repositories though?
Develop on Gentoo and ebuild comes almost free.
Then put your packaging effort into getting into Debian. Consider that good enough. All the users of Debian based distros can just enable ‘universe’ to install your package.
Then maybe add in Arch if you feel generous.
That ought to cover it. Anything else is either so niche that it can take care of itself or it’s RedHat and F@# RedHat!
Yes, because that being possible and practically feasible means there’s less friction in getting it to yours. That “got mine, f— you” approach is what balkanization is all about, and what causes all the trouble even if you do happen to sit in the largest camp.