Disclaimer: I wrote this article and made this website.
There was some talk of this issue in the recent fediverse inefficiencies thread. I’m hopeful that in the future we’ll have a decentralized solution for file hosting but for now I deeply believe that users should pay for their own file hosting.
am I the only one who would advocate for text only storage: no. This comment also gets my point? There should be text only lemmy instances which only save text and do not allow any kind of image posting or storage.
- text is less offensive to consume and moderate than evil images or video
- text is way more information dense and can be even compressed more! Truly the green biosphere friendly data format. I would be willing to save text only data of strangers on my hard-drive, but not images or video. Could even be valuable llm analysis training data.
Yes people could post base64 encoded images, but that is a larger technical barrier and can be detected. If image storage is really need, images should be heavily compressed (webp 90% quality loss), provided as links to external sites, and whenever possible svg / vector graphics should be preferred.
Have you considered providing something like this: https://jortage.com/ and maybe contribute to their efforts to develop a specific API for that? Source code is here: https://github.com/jortage
Jortage is a really interesting approach. It definitely helps reduce the impact of the file hosting problem but it doesn’t fully address the underlying cost issue. The cost of storing files grows every month indefinitely while donations typically don’t.
I would like to see a file hosting pool come to lemmy though. So I will look into it. :)
I think the major advantage is the deduplication - when an image goes viral across Mastodon (or Lemmy) it’s currently stored hundreds or thousands of times, each with its own cost. Do you dedupe (for either your customers’ benefit or your own)?
Are the images duplicated when shared? My understanding is that only a link to the file is replicated across servers and duplication comes from users manually uploading the same file to another server.
My website does not do any deduplication at this time.
Yes, for example go to https://infosec.exchange/explore
I see the top post as https://infosec.exchange/@nocontexttrek@mastodon.social/113433063621462027 and the image is https://media.infosec.exchange/infosec.exchange/cache/media_attachments/files/113/433/063/582/671/258/original/71da3801e4e4f08c.png
The link is to the original on https://files.mastodon.social/media_attachments/files/113/433/062/676/773/993/original/f828afef5cc7ed1c.png but if you click the javascript loads a modal with the local cached version (same image as the thumbnail that infosec.exchange loads.
There’s lots of different codebases across the fediverse so perhaps some hotlink, but local copies is the default.
The Lemmy server config indicates that is an optional setting to improve user privacy so requests don’t ever hit the original server from the client. Those cached files are only temporary and will be deleted after some time. So it’s not really full blown duplication.
The default setting is to only generate the thumbnails and store those locally (indefinitely?) but even that can be turned off. I checked and it appears that lemmy.world has the thumbnail generation disabled so all images from other instances just link to the original on that instance.
Ok, so Lemmy doesn’t cause the same amount of duplication, but I’d still argue that dedupe is valuable: it saves on hosting costs (your costs, in this case) and users will get a small advantage in having slightly higher cache hits.
For sure, I’ll add it to the list. :)
Pict-rs that used by Lemmy to store images already supports S3 type storage, so in theory it should work with Jortage, but I don’t think anybody has tested that yet. The people behind Feddit.org might have experimented with it as they expressed interest a while back.
Interesting approach, good luck! Admittedly I’m not sure if many users want to take their media uploading in their own hands and pay for it but maybe I’m wrong. Where are the images stored? Do you have your own hardware? Backups etc?
Also since you’re interested in Fediverse media storage, I recently read about https://jortage.com/ It’s a third party storage for your instance with deduplication, pretty interesting idea. Takes away a bit of the federated part though
I know I’m not necessarily the target audience for this. But it feels too expensive. 6x the price of Cloudflare R2, almost 13x the price of Wasabi. Even iCloud storage is $0.99 for 50 GB with a 5 GB free tier. But again, I know I’m not necessarily the target audience as I have a lot of technical skills that maybe average users don’t have.
If you ever get around to building an API, and are interested in partnerships, let me know. Maybe there is a possibility for integration into !echo@eventfrontier.com 😉.
Yeah, I wish it could be cheaper but I’m not a corporation. Instead I’m dependent on them to make a simpler product.
The target audience is certainly not developers because they can jump through the hoops to setup their own S3 + CDN or similar.
I found this https://github.com/Marie673/Torrent_Proxy
Its in chinese so idk if its whats needed but if each instance also acted as a torrent proxy then thats decentralised domain agnostic file hosting that doesnt break frontends but also allows clients to update to do resource resolution themselves while helping to serve the fediverses files as a whole.
IPFS?
as I stated in this comment it’s not really feasible as to ~5s delay that was tested some time ago.
That’s the wrong comment.
What would an IPFS solution look like here? That’s a genuine question. I don’t have much experience with IPFS. It seems like it isn’t really used outside of blockchain applications.
The sustainability of it is questionable. If I’m not mistaken, IPFS is based on Ethereum, which has gone over to proof of stake rather than proof of work, but it’s still a pretty cumbersome system.
We’re talking about something that needs to compete with Quic and CloudFlare. I’m not sure that Ethereum or even crypto itself is efficient enough as a content delivery method, that IPFS - though a nice idea - is unrealistic.
But that’s just speculation from someone who has zero knowledge behind IPFS as a technology and protocol, so take it with a grain of salt.
IPFS has absolutely nothing whatsoever to do with Ethereum, or indeed any blockchain. It is a protocol for storing distributing and addressing data by hashes of the content over a peer to peer network.
There is however an initiative to create a commercial market for “pinning*”, which is blockchain based. It still has nothing to do with Ethereum, and is a distinct project that uses IPFS rather than being part of the protocol, thankfully. It is also not a “proof of work” sort of waste, but built around proving content that was promised to be stored is actually stored.
Pinning in IPFS is effectively “hosting” data permanently. IPFS is inherently peer to peer: content you access gets added to your local cache and gets served to any peer near you asking for it—like BitTorrent—until it that cache is cleared to make space for new content you access. If nobody keeps a copy of some data you want others to access when your machines are offline, IPFS wouldn’t be particularly useful as a CDN. So peers on the network can choose to pin some data, making them exempt from being cleared with cache. It is perfectly possible to offer pinning services that have nothing to do with Filecoin or the blockchain, and those exist already. But the organization developing IPFS wanted an independent blockchain based solution simply because they felt it would scale better and give them a potential way to sustain themselves.
Frankly, it was a bad idea then, as crypto grift was already becoming obvious. And it didn’t really take off. But since Filecoin has always been a completely separate thing to IPFS, it doesn’t affect how IPFS works in any way, which it continues to do so.
There are many aspects of IPFS the actual protocol that could stand to be improved. But in a lot of ways, it does do many of the things a Fediverse “CDN” should. But that’s just the storage layer. Getting even the popular AP servers to agree to implement IPFS is going to be almost as realistic an expectation as getting federated identity working on AP. A personal pessimistic view.
TL;Dr. From Wikipedia
IPFS allows users to host and receive content in a manner similar to BitTorrent. As opposed to a centrally located server, IPFS is built around a decentralized system of user-operators who hold a portion of the overall data. Any user in the network can serve a file by its content address, and other peers in the network can find and request that content from any node who has it using a distributed hash table (DHT).
So it’s BitTorrent in the web browser… thanks. How is that to be competitive with CloudFlare and Quic again? It has the same network issues that the blockchain has, in that it will be cumbersome and slow - for anyone else that doesn’t have millions to throw into infrastructure. Welcome to the same problem again, but in a different way.
Ironically, because there’s no UDP in browsers, we can’t actually get proper p2p on the web. WebRTC through centralized coordination servers at best. Protocol Labs has all but given up on this use-case in favor of using some bootstrapped selection of remote helper nodes.
Is a p2p system for media with the instances just hosting magnet links too slow for fediverse purposes? To me this seems like the most resilient way to handle media in a decentralized system
If a social network is to take off, it must be accessible from mobile devices behind CGNAT (carrier grade network address translation).
Why?
Because that’s where all the users are. The “social” aspect of a “social” network. Anyone can host a forum but it’s useless without users.
I think Tenfingers could be an interesting option as hosters do not know what they host, the data can be modified, and it’s 100% decentralised.
Thank you for writing this. Small typo: focued (focused).
Thanks for reading and pointing out that typo! (I fixed it)
Ok, hear me out.
We find the users with the slowest internet and start sending them all the data. They don’t have to keep anything on disk. Then they send it all back and forth between each other. Any time a user makes a request, we just wait for one of the slow nodes to come across the data and send it out.
We use the slowest wires for all the storage. It’s fool proof.
You jest but… delay line memory
This is amazing and I love it.
Top 5 YouTube videos this one
Somebody actually did make this as a joke years ago haha https://github.com/yarrick/pingfs
I was brushing my teeth when reading this comment and inadvertently ended up swallowing all my toothpaste.
don’t forget to spit
Ha! That’s awesome!
You’re not the first to think about this.
See https://aumetra.xyz/posts/the-fedi-ddos-problem - there an embed server is proposed, to be shared by multiple instances (ideally a great many would use just the one), which can host things like image files and previews.
There’s a big issue with this.
If malicious content like CP gets uploaded on to a server, obviously other servers do not want this to be replicated to their servers. So how would you solve this problem? Well they could give all moderation power to the original server they’re replicating, but that could be far too slow or they could even miss malicious content like this. Or maybe they even disagree about taking down certain things.
Another solution is that any server participating in the content mirroring could take it down for just themselves or for all the other members as well. The issue here is now you’re expanding moderation abilities, while also giving the other servers much more responsibilities.
It’s not as simple as wanting to replicate content. If you host it, you are responsible for any illegal content a user may upload to it. Not to mention laws vary by country as well. Ignoring the technical challenges here, it’s also mandatory that the other servers replicate the other servers data to also choose to be responsible for what gets uploaded. And that is a really big ask. The law doesn’t care about the technical reasons, they’ll just see illegal content uploaded to your server.
I think IPFS could help the fediverse with storage.
I expect something like this to end up being the solution, but I think we’re far from a consensus in that regard.
I think most architecture design decisions are made by the developers of the fediverse projects. If the 3 Lemmy devs or the Mastodon maintainers agreed to do it… (And it’s technically feasible.) I suppose it could be done.
I mean as long as it works seemlessly and doesn’t violate ActivityPub, we don’t really need a consensus of all the users and admins. We just need the server admins to install the next update.
To actually keep data persistent on IPFS and not be deleted by the garbage collector, you need to have a server(s) pin the node that holds that data.
You either host these servers yourself, or pay providers to store it for you.
And at that point you just reinvented a server simply hosting your data but with extra steps.
Thank you for pointing that out. I’m not familiar with IPFS but I tend to agree there’s no free lunch here. People think you can wave the blockchain wand and free computing appears but there’s always costs built in somewhere.
This feels like something the Fediverse is ultimately going to build for itself. I know jack squat about the details, but it’s gonna have to be a thing eventually, I think.
What is stopping some big giant, let’s say Yahoo/Verizon from buying a shitload of storage, starting their own private instance which is open to the public, but private in the sense that only Verizon employees are admins and mods. Only Verizon controls things. Then advertise to the point that the average person on the street knows that Verizon.Lemmy exists, and assosiates Lemmy with being a Verizon thing? What is stopping big tech from pouring the money required for this concept to take off, and using their control over their instance from making the decentralized a centralized service in the general public’s minds?
Right now Lemmy is 60k people. Ok. What if Lemmy was 200 million people, and only 60k knew it was a decentralized service? Everyone else just thought Verizon owned Lemmy?
Either they federate and all their users are exposed to the rest of the fediverse, or they don’t and they may as well be a separate thing
Yeah. What I’m saying is, they federate, but people have no idea what “federate” means. So they’d come here, and see “@smeg@feddit.uk” and not understand what feddit.uk was.
They would see you, and think you are a user of the verizon owned service. Not question it one bit, and just move on thinking it’s all verizon.
The same way people in Atlanta will say “I want a coke” “What kind of coke?” “Root Beer”.
Or the same way parents in the 90s would say “I bought you a Nintendo Game!” then you open it, and it’s a Sega Saturn disc, when you have Sony Playstation. It’s all just a Nintendo to them.
I’m saying if Verizon grew Lemmy to 200 million users, and all except 60k were on the Verizon instance, then despite being incorrect, Lemmy becomes “The Verizon owned Facebook”.
Doesn’t matter that it’s federated.
Isnt this the exact reason why there was such concern over the idea of Threads federating with the fediverse at large?
This is literally exactly what happened to email. It didn’t go great
I wish there was some version of PBS for Lemmy, like public funds for hosting. I’ll admit I haven’t really thought this through, so there’s probably some problems with my idea.
At least as far as US law is concerned, a federally hosted and administrated social media platform gets interesting with America’s unusually strong free speech laws, since there’s content which is legal but unethical which they likely would not be allowed to block or moderate, such as bullying, hate speech, misinformation, etc. but also illegal content would be immediately moderated away, which might include content that falls into legal grey areas or ethical but technically illegal content, like someone copy/pasting the contents of a paywalled article, or discussing any kind of DRM or digital security bypass
Honestly I think there’s good reason for governments to host a Mastodon instance for their representatives to use for communications, but inviting the public to use it might get weird for sure
