Hacker News

Show HN: Ratarmount 1.0.0 – Rapid access to large archives via a FUSE filesystem

Hacker News - Fri, 11/01/2024 - 11:25am

Hi HN,

Since my first posted introduction of ratarmount [0], 2 years have gone by and many features have been added.

To summarize, ratarmount enables working with archived contents exposed as a filesystem without the data having to be extracted to disk:

pip install ratarmount ratarmount archive.tar mounted ls -la mounted I started this project after noticing the slowness of archivemount with large TAR files and wondering how this could be because the file contents exist at some offset in the archive file and it should not be difficult to read that data. Turns out, that part was not difficult, however packaging everything nicely, adding tests, and adding many more formats and features such as union mounting and recursive mounting, are the things keeping me busy on this project until today. Since the last Show HN, a libarchive, SquashFS, fsspec, and many more backends have been added, so that it now should be able to read every format that archivemount can and some more, and even read them remotely. However, performance for any use case besides bzip2/gzip-compressed TARs may vary even though I did my best.

Personally, I am using it view to packed folders with many small files that do not change anymore. I pack these folders because else copying to other hard drives takes much longer. I'm also using it when I want to avoid the command line. I have added ratarmount as a Caja user script for mounting via right-click. This way, I can mount an archive and then copy the contents to another drive to effectively do the extraction and copying in one step. Initially, I have also used it to train on the ImageNet TAR archive directly.

I probably should have released a 1.0.0 some years ago because I have kept the command line interface and even the index file format compatible as best as possible between the several 0.x versions already.

Some larger future features on my wishlist are:

- A new indexed_lz4 backend. This should be doable inside my indexed_bzip2 [1] / rapidgzip [2] backend library.

- A custom ZIP and SquashFS reader accelerated by rapidgzip and indexed_bzip2 to enable faster seeking inside large files inside those archives.

- I am eagerly awaiting the Linux Kernel FUSE BPF support [3], which might enable some further latency reductions for use cases with very small files / very small reads, at least in the case of working with uncompressed archives. I have done comparisons for such archives (100k images a 100 KiB) and noticed that direct access via the Python library ratarmountcore was roughly two times faster than access via ratarmount and FUSE. Maybe I'll even find the time to play around with the existing unmerged FUSE BPF patch set.

[0] https://news.ycombinator.com/item?id=30631387

[1] https://news.ycombinator.com/item?id=31875318

[2] https://news.ycombinator.com/item?id=37378411

[3] https://lwn.net/Articles/937433/

Comments URL: https://news.ycombinator.com/item?id=42017833

Points: 4

# Comments: 0

Categories: Hacker News

Ask HN: Has the LLM/transformer architecture hit its limit?

Hacker News - Fri, 11/01/2024 - 11:23am

Have we hit the limit for performance increases on the current architecture of LLMs?

I’ve heard some amount of agreement among professionals that yes we are, and with things like papers showing Chain of Thought isn’t a silver bullet it calls into question how valuable models like o1 are it slightly tilts my thinking as well.

What seems to be the consensus here?

Comments URL: https://news.ycombinator.com/item?id=42017814

Points: 2

# Comments: 6

Categories: Hacker News

Universal Paperclips

Hacker News - Fri, 11/01/2024 - 11:20am
Categories: Hacker News

Writes and Write-Nots

Hacker News - Fri, 11/01/2024 - 10:40am
Categories: Hacker News

Show HN: TotalPause – an app to suggest breaks and promote mindfulness

Hacker News - Fri, 11/01/2024 - 10:36am

Hello, I created this macOS app because I often spend long hours at my computer without taking breaks. Although I've tried many similar apps, few really meet my needs. For instance, Pomodoro timers feel draining and don’t suit my work environment. Other apps often trigger breaks at inconvenient times, becoming more annoying.

I decided to develop my own app to try to address my own issues. My day job is as a backend engineer, primarily working with distributed systems and DevOps. I have knowledge of web development but had never developed a desktop app before. I started learning and building TotalPause in my free time last year. There is also a blog post[1] about why and how I built it.

The goal with TotalPause is to make it as unobtrusive as possible, requiring minimal interaction, making it appear less frequently as work habits improve, and avoiding the drawbacks of full-screen break reminders through technical implementations like auto-focus mode (automatically detecting applications, meetings, and games), activity analysis, and maybe other advanced features.

Hope to see if it works for others with similar needs. Thanks.

[1] https://blog.damnever.com/en/2024/reminding-myself-to-take-a...

Comments URL: https://news.ycombinator.com/item?id=42017323

Points: 1

# Comments: 0

Categories: Hacker News

Ask HN: Beginner Blogger Looking for "Proofreaders"

Hacker News - Fri, 11/01/2024 - 10:33am

Hello everyone, I'm hosting my own blog and starting to write. However, I don't feel confident publishing without having at least a few people's opinions.

Considering many HN users are often reading blogs, articles or essays I thought it would be a fine place to ask for help. I would like to get in touch with some people that would kindly proofread my work and give honest opinions. My first article is only a 5-minute read but after receiving opinions I might rewrite it or, who knows, discard it.

Anyway, I'm a noob writer asking for help from fellow readers or writers!

Please get in touch with me via my email address (see my profile).

Comments URL: https://news.ycombinator.com/item?id=42017281

Points: 1

# Comments: 0

Categories: Hacker News

Pages