So first off: Very cool. Amazingly polished, and self-hosting! Especially since it looks to be an actually-independent project with POSIX as an optional compat layer.
I feel like the features that are unique are mostly in 3 groups:
1. Features that shouldn't be unique to this system. Tabbed windows have occasionally happened on other systems, and probably should become more common, and there's no reason it shouldn't work on other systems. I hope this inspires other people to copy;)
2. Features that only work if you have really tight integration or different primitives than most OSs are using. Programs updating file name when you rename from the file manager, or the file manager showing when files are open... might be possible to graft on to other systems, but it'd be a right pain. Even systems like MacOS are going to struggle - the whole OS is under one company that could do cross-functional stuff like that, but any time it touches applications you need external developers to support it and that may or may not work out.
3. Features that are possible, but nobody else does it because it's impractical - possibly only impractical from their starting points, though. Showing total size of subdirectories is expensive on ex. Linux because you have to walk every file recursively. I don't know if they're just eating the cost because they haven't hit a case where it matters, or if their system actually makes it cheap (I could easily imagine a filesystem that moved the calculation cost up front).
Your examples of (2) and (3) have nothing to do with the desktop, really, or, rather, they are about things that one could see reflected in the desktop if only they were feasible at much lower layers.
"total size of subdirectories" is basically impossible as long as there are hardlinks. But hardlinks are useful, so. Even w/o hardlinks, if you have anything like ZFS snapshots and clones then you have multiple kinds of "total size" to show, like "total size" of all the files below vs. total space consumed by those files in this snapshot/clone. You could have hardlinks and all of them and then asynchronously update caches of sizes, but now you waste cycles on that and those cached sizes will often be wrong. You basically can't get there without making trade-offs that kinda suck. It's just better to not bother with this particular feature.
You're assessing the feasibility here making the assumption that a polling pattern would be used but this is exactly what an event based pattern is perfect for. As long as files are modified through the os you can propagate changes upwards and only update folder metadata when required. This is what the parent meant by moving the cost upfront.
If your only option is polling you're right but we're talking abou an os from first principles here. You have the control to make sure all file changes go through the proper api and update the metadata to be as accurate as you like.
There's a lot more to it than that. For example, two links to a file in the same directory shouldn't count as doubling the space it consumes in that directory nor all its parents -- that's an easy case to consider, but there are others that are much harder.
Adding different kinds of measurements of size doesn't particularly increase the difficulty of tracking it here, just adds some more fields to the metadata and maybe some additional events.
To take an example of size data types from another thread, let's imagine you want to track 2 kinds of size: size as read sequentially (A) and size gained if deleted (B). The first cares about links but the second does not. Every folder would track the size of sub-directories by listening for size calculation events firing on the contained files and folders. A lowest level folder would listen to the events of all its files and update it's own size metadata, this triggers another event that any parent folder is listening to, repeat. A link would just listen to the event of its linked file or folder and would only indicate size updates as relevant to the size data type A. This information gets propagated upwards normal.
As you say, there's a lot of details here but I'm not coming up with any blockers that would invalidate the pattern. It's a very flexible system and I've written this sort of propagating update before with future features all being simple to add. If you can think of any blockers I would be very happy to hear them so I can design around it next time I touch a project like this.
This is exactly how I assumed modern operating systems would operate (macOS seems to operate this way because you can actually show size subtallies, but it actually polls and caches; Windows doesn’t even bother unless you get properties on a directory, which is lame; I think beOS may have actually done this?), and was disappointed when I realized it wasn’t… and still doesn’t for some reason? Why not? Couldn’t they just route all OS-level ways of writing to the storage medium through a system like this?
There's no real reason they couldn't but polling systems tend to be the first thing implemented as conceptually they are simpler and require less rigor to enforce. Switching from a polling pattern to an event pattern however is a sizeable amount of work required and you can frequently get bugs in the switch. The discussion here is in the context of a greenfield project where such concerns aren't an issue.
To be clear it's not like an event registration system is better in every way than polling. It's trading some extra hd and (potentially) memory usage for upfront and cheaper cpu costs. It is possible that whoever made the decision weighed the two options and decided on polling instead of events, and this decision was made long ago when os filesystems were first being designed and storage was at much more of a premium.
I don't see why "size gained if deleted" "does not [care about links]".
What I've learned in my experience with ZFS, Lustre, and other filesystems is that the user simply cannot get what you're asking for, not with any kind of real reliability. For distributed filesystems (like Lustre), the kind of thing you're asking for is simply ETOOHARD or ETOOSLOW. It's very easy to insist on a solution, and very hard to get one.
> I don't see why "size gained if deleted" "does not [care about links]".
Deleting links does not give you any additional storage space beyond the minimal amount taken up by the link itself.
As for existing filesystems, yeah they're going to have problems as they're built on filesystems that for the most part are polling. I'm not insisting on a solution or saying existing filesystems need to use this though, this is all in the context of a greenfield project. Switching something like linux to use an event based system instead would be a major project.
As for distributed filesystems where every file does not own all its own bits, that's just a different way of measuring and doesn't have any major problems for the concept.
I think the vector based UI should be on the list somewhere. Really smart solution to the HiDPI problem that gives you easy arbitrary scaling without any awkwardness
Low-resolution monitors and laptops remain widespread [0], and vector-based UI just doesn't look that good on those (either blurry or misaligned). I'm doubtful that HiDPI will become the mainstream standard anytime soon outside of smartphones and tablets (small screens). Cost increases quadratically with screen resolution as a function of production yield, so low-DPI hardware is likely to remain the cheaper option that many people will continue to choose.
Agreed that it’s a safe thing to say, but there’s nothing stopping an OS from putting in the work to do that, and therefore allow them to have a single vector-based UI for Low and Hi DPI screens.
Right, but e.g. SVG doesn’t support hinting AFAIK. Also, realistically nobody is willing to do the work to add hinting to vector icons. Fonts with good low-res hinting have already gotten rare.
I suspect that we’ll continue to live in a world where the UI is neither optimal for HiDPI nor (any more) for low DPI, while really high DPI (e.g. over 200 DPI on desktop monitors), which would enable going full vector without detriment, will rather remain the exception than becoming the rule.
They have, no? GNOME/KDE have supported SVG icons for years, the themes are usually vector based. I don't think there are necessarily many bitmaps in a default modern GNOME install.
For macOS the reason is simple and deliberate: it's much easier for artists to make beautiful icons as bitmaps. They can use Photoshop and not just vector editors. Given that screen resolutions increase at a somewhat slow and predictable pace it means they can just redraw icons at higher resolutions from time to time to keep up with their hardware changes. They like to change art styles anyway.
Well, they don't have it in such a way that things are arbitrarily scaleable (although things are slowly moving in that direction). If you look at the demo video you can see the entire GUI scale right up and down in real time
Like, I almost want to say that vector should be the native format for anything created on a computer, even in "paint" programs (which might for example cache a rasterized version too)
Weren't both X11 and Windows originally vector-based (as in drawn using lines, rects, and fills) UIs, but everyone moved to pixmaps because the X11 protocol didn't support anything remotely SVG-like and client-side drawing was faster?
What’s the state of the art for file systems in the age of M2 drives that are bonkers fast? I dunno about everyone else, but knowing hierarchically where my disk space is going is a really common concern.
I would imagine, the “m2 fast” is visible more in reading medium sized files than 100k tiny reads (when recursively querying directories like this case GP is taking about)
To make GP’s use case better, FS really needs an custom index or require that every write also update all hierarchy. Unless of course they work like indexers and just work with delayed data
Good point but likewise one has to wonder why the need for folder hierarchies anyway. With bonkers fast disks and cpus the organising of files should be able to be dynamic and driven by some form of metadata and tags rather than static tree structures. The web doesn’t first require you to define a structure before search so why should a desktop?
This is exactly what Steve Jobs would tell me every week when he would look at the latest Finder build. Of course, we needed multiple pieces of infrastructure to get there; a journaled and indexed filesystem, kernel-level, file system modification notifications, metadata indexing systems, etc. All of the pieces did appear, but there are also those darn stubborn users who kept insisting on the ability to navigate a physical file system to an actual file.
Many of the features of OSX (which I rarely use, but worked tirelessly on) came as a result of Steve's dislike of having to know where files were and having to navigate to them. The Dock, Spotlight, Mission Control, Expose, even Time Machine all came out of Steve's hard-to-pin-down concept of what a modern user interface should be.
I suspect it may be possible to create a fully "Smart Folder" query driven Finder experience. Your sidebar could be populated entirely of saved queries. This might be a fun experiment!
So I'd have to tag every one of the umpteen files I create (in some session doing whatever) with some spur-of-the-moment searchword I'll never remember later, in stead of just cd-ing to some directory and that's where they'll all be?
It's funny how all these anti-directory tag-and-search proponents seem to take for granted that their hobbyhorse is obviously superior. I've never seen any proof that it is, and I don't think it is.
No, I just create files. I don't have to do any tagging at all.
Or: Yes, in a way that's what the hierarchical directory structure already does for us.
In either case: So what's the use of ripping out the hierarchical directory structure and replacing it with some (other) "tagging" system???
(We already have a universal "tagging" system that automatically uses every single word in every single file as a searchable "tag"; it's called "grep"...)
Because spacetime is hierarchical in that way too. I can put a thing in a box in a room, but I can't put the thing in multiple boxes in various rooms. The filesystem thus makes intuitive sense: we store physical objects in exactly the same way.
Now sure, it'd be nice to grep through my house, but I still couldn't hang the same painting in two places at the same time.
On a related note, why a filesystem and not a database? I get the feeling that a filesystem ultimately is just a .. poor man's database. Except that it's quite crappy and there are no proper transactions, many things are impossible without TOCTOUs, many a bug (including security ones) have been due to race conditions around filesystem operations.
I guess path->{metadata,data} with no transactions is a simple abstraction, as with so many other simple abstractions, it just bites you when you try to build anything nontrivial. Then you need to switch to a real solution. Just as people sometimes start with a bunch of shell scripts before they recoil in horror and realize that they should've started with a real programming language.
ProtonMail launched with labels only. They later added folders for those of us who like to file messages under "organized now, don't want to think about this again".
Folders are so common still in part because many of us want them to the point that both Fastmail and Gmail lets you use tags in ways that behaves almost entirely like folders, including a hierarchy.
I'd never consider an e-mail system (or a filesystem) that doesn't allow me to at least very closely approximate that structure.
> Showing total size of subdirectories is expensive
To me the real obstacle isn't that it's expensive (I mean, you could ostensibly cache the size on disk for each directory, if not find some other clever algorithmic solution), but that the notion of "size of a directory" itself isn't all that meaningful in the presence of e.g. hardlinks.
Roughly 0% of a typical desktop’s disk space is used by hardlinked files. You can safely double count them. That’s exactly what every disk space analyzer does!
If you really wanna avoid double counting, just divide the size of every file by st_nlink. Of course you’d have to update the cached sizes of every directory that has a link to that inode so you’d need to cache the mapping from inodes to paths too. Another solution is to cache 2 sizes per directory, one for all files with 1 link and another for files that have 2 or more. The UI could hide the latter when it’s 0GB. But this discussion is academic; Nobody really cares about hardlinks.
> Roughly 0% of a typical desktop’s disk space is used by hardlinked files. You can safely double count them. That’s exactly what every disk space analyzer does!
While that may be a reasonable strategy, it's not what every disk space analyzer does.
I've got a backup setup that uses hardlinks to provide a wide variety of restore points without using a lot of space. du doesn't double count:
$ du -hs daily.0
436G daily.0
$ du -hs daily.1
436G daily.1
$ du -hs daily.0 daily.1
436G daily.0
12M daily.1
Not sure what you consider a "typical desktop", but on Windows, WinSxS has gigabytes worth of hardlinks. If you don't care about them that's another matter I guess.
Also note that the user will be confused when they delete the whole directory and observe 0 bytes get freed. (I guess a similar problem is also there even if you double count.)
The point is, the problem itself is-ill defined. There's no solution to that other than scrapping or redefining the problem itself. And it's hard to define the problem precisely for a non-technical user.
Then you get into semantic arguments: If a directory contains 2 1GB files, does the user care that 99% of their blocks are shared, or that just an under-the-hood implementation detail, and the user wants to know that there are 2GB worth of files in there?
Really, there should be two file sizes: "how much space this will take if I copy it to other filesystem" and "how much space will be freed if I delete this"
Well, three, "how many bytes do I get if I open it and read all the bytes out". Or maybe four, how many bytes do I get if I open it and read all bytes which aren't holes (ie. how many bytes do I need to put into an archive that supports sparse files) :)
I would expect at least one of those cases to be identical to "how much space this will take if I copy it to other filesystem" if you're asking a generic question where the target is a hypothetical and therefore you have to say how many bytes would be taken by the raw files since everything else requires knowing specific details about the target.
> "how much space this will take if I copy it to other filesystem"
This is ambiguous between "how many bytes are all these files in total" and "how many bytes does it take to store a single copy of all these files on such-and-such file system (mostly the current one)". The latter can be different because of transparent compression, which is common on e.g. BTRFS.
> Features that only work if you have really tight integration ... Programs updating file name when you rename from the file manager ... Even systems like MacOS are going to struggle
MacOS has actually had this one in the bag for a couple of decades now (likely due to their really tight integration).
Doesn't even Windows do that? Rename a file in File Explorer, and the app that has it open pops up a dialogue saying "File So-and-so has been renamed.", or some such. But maybe that's because the app has to do the work of keeping track of it (though probably via hooks into the OS / Explorer), so it's not fully automatic? Can't recall if all apps do it, think I've only noticed it in a few.
Again, it depends on how the application accessing the file is developed.
If your program just read to copy in memory then closes the file, waiting for the next save to reopen it, the OS can’t know on which file you are working. And this, i think, must be a pretty common implementation.
You can use inotify to figure out what files are opened, closed or when they are written too. Can be used for example to indicate active file downloads in the file manager.
However I don't think that's quite enough, as most apps will close the files again after they have read them. There is no need to keep them open, as you are working on their in-memory representation, not on the file on the disk until you hit 'Save'. So this seems more like a desktop level feature similar to "Recent Files", where the apps has to manually announce what it is doing instead of relying on low level file operations to figure it out.
If every application uses inotify and correctly handles the events, I expect so. Which brings us back to "even if you can do it good luck getting buy in".
> Programs updating file name when you rename from the file manager, or the file manager showing when files are open... might be possible to graft on to other systems, but it'd be a right pain. Even systems like MacOS are going to struggle - the whole OS is under one company that could do cross-functional stuff like that, but any time it touches applications you need external developers to support it and that may or may not work out.
I thought Mac OS apps could and did do just that. Perhaps it's an inconsistent behaviour that depends on the developer implementing it rather than something that comes for free by using Swift UI or Cocoa.
I think the point of the tabbed windows isn’t so much window tabbing, as the idea that the window belongs to the user and not the app.
The user manages windows and not apps.
Currently, this is being leveraged in tabbed apps, and in the ability to open a window, resize it, etc and then decide which app should show itself in that tab of that window.
But the paradigm change itself could potentially lead to new workflows, etc if creative app developers and designers really start working woth it.
I feel like the features that are unique are mostly in 3 groups:
1. Features that shouldn't be unique to this system. Tabbed windows have occasionally happened on other systems, and probably should become more common, and there's no reason it shouldn't work on other systems. I hope this inspires other people to copy;)
2. Features that only work if you have really tight integration or different primitives than most OSs are using. Programs updating file name when you rename from the file manager, or the file manager showing when files are open... might be possible to graft on to other systems, but it'd be a right pain. Even systems like MacOS are going to struggle - the whole OS is under one company that could do cross-functional stuff like that, but any time it touches applications you need external developers to support it and that may or may not work out.
3. Features that are possible, but nobody else does it because it's impractical - possibly only impractical from their starting points, though. Showing total size of subdirectories is expensive on ex. Linux because you have to walk every file recursively. I don't know if they're just eating the cost because they haven't hit a case where it matters, or if their system actually makes it cheap (I could easily imagine a filesystem that moved the calculation cost up front).