You say this as if it is an original idea. Of course the IA is working on this and have been for over 6 years. There already is a DWeb version. They have been advancing DWeb infrastructure. The IA hosts all kinds of DWeb developer events.
But it is over 50 petabytes and the IA gets a huge amount of traffic through the regular web that they need to serve quickly and efficiently to their users.
Guess what has happened over 6 years of decentralization of 50 TB? People only seed what they want or care about and there aren't enough seeders to host. They set all this up and nobody volunteers. You're a DWeb advocate and you haven't been seeding. That's a recipe for disaster if they rely on the goodness of volunteer seeders. The IA's mission is broader. DWeb will ever only compliment the IAs mission.
Incentivising seeding is hard. Maybe cryptocurrencies can be useful here, but I understand not everyone likes them especially here on HN. In retrospect the ideal setup would have been if archiving was included into the core HTTP protocol.
How about you leave this to the people who know what they are talking about? Your wild ideas are not contributing to this discussion.
Edit: When a hacker has what they think is an original idea and hears that people have been working on it for years, the hacker's response should be joy and a desire to dive deep into that prior work. You instead ignore all that work and think that whatever popped into your head after two minutes is a solution.
I kind of agree, but the way the internet is going, with everyone being behind carrier-grade nat, it's not much of a decentralized network of computers anymore, not to mention all the kids with their laptops and tablets not even hosting anything :(
UPnP is just automating the process of forwarding ports, CGNAT will still screw you sideways because you're behind a router can't access or order around.
I don't know if bittorrent has improved - but 20 years ago I had a personal issue with it.
At that time our son was using it for games. He goes away to college and came home for the first school break. I get a phone call from our internet provider asking if our son was home. I was so shocked and handed the phone to our son.
Apparently at that time bittorrent was optimizing for the most efficient path to a host. Since we had relatively good connection, the mighty weight of the internet was funnelling through our tiny internet provider to our son's computer. The provider (without our knowing it) had made a deal with our son that he would only turn on bittorrent between midnight and 6 AM. I doubt other providers would be so generous.
I have been sceptical of bittorrent since that day.
All clients today (and probably back then) have options to limit bandwidth consumption including throttling, scheduling, and total data transfer caps. For serving mostly HTML and images, dedicating even 10% of a home broadband connection to serving content would allow many, many people per day to access archived pages.
They black out all items that get a DMCA complaint or similar request (so it's still there just not accessible). However they permanently delete illegal stuff.
I would assume they delete illegal stuff as they are compelled to. What I'd like to know is their policy for legal stuff that they exclude that is not as a result of DMCA.
I thought Cloudflare was going to provide "Always Online" access to Internet Archive
https://blog.cloudflare.com/cloudflares-always-online-and-th...
Other way around! Cloudflare can optionally load your site from IA if it's down.
That is about us using the Internet Archive to show a snapshot of a page from the Internet Archive when the origin server is down.
Still down in my town.
It would be better if the Internet Archive was decentralized without a central point of failure, maybe run on something like bittorent.
You say this as if it is an original idea. Of course the IA is working on this and have been for over 6 years. There already is a DWeb version. They have been advancing DWeb infrastructure. The IA hosts all kinds of DWeb developer events.
But it is over 50 petabytes and the IA gets a huge amount of traffic through the regular web that they need to serve quickly and efficiently to their users.
Guess what has happened over 6 years of decentralization of 50 TB? People only seed what they want or care about and there aren't enough seeders to host. They set all this up and nobody volunteers. You're a DWeb advocate and you haven't been seeding. That's a recipe for disaster if they rely on the goodness of volunteer seeders. The IA's mission is broader. DWeb will ever only compliment the IAs mission.
https://blog.archive.org/2021/02/18/behind-the-scenes-of-the...
https://www.bleepingcomputer.com/news/technology/archiveorg-...
If I help seed this DWeb and it turns out it has some copyrighted materials in it, will I be potentially held liable?
It is not my original idea, but it is an obvious idea to anyone who knows how the internet works, I just added it to get the discussion going.
It already exists. IA has had a DWeb for 6 years. Nobody seeds it.
Incentivising seeding is hard. Maybe cryptocurrencies can be useful here, but I understand not everyone likes them especially here on HN. In retrospect the ideal setup would have been if archiving was included into the core HTTP protocol.
How about you leave this to the people who know what they are talking about? Your wild ideas are not contributing to this discussion.
Edit: When a hacker has what they think is an original idea and hears that people have been working on it for years, the hacker's response should be joy and a desire to dive deep into that prior work. You instead ignore all that work and think that whatever popped into your head after two minutes is a solution.
The problem is that it's hard to do this in a way that ensures good archival of ALL resources.
Bittorrent works well for popular things but fails for marginal content (unless some really dedicated individuals step in.)
What the internet archive provides is a way to have access to many many resources which you didn't know you needed in advance.
I kind of agree, but the way the internet is going, with everyone being behind carrier-grade nat, it's not much of a decentralized network of computers anymore, not to mention all the kids with their laptops and tablets not even hosting anything :(
UPnP exists and allow devices to ask the router to open a port to them.
UPnP is useless with CGNAT (Carrier Grade NAT), which is what the op is talking about.
There are other ways to get seeding working, though, including IPV6, which is gaining adoption, so I don't agree with the OP.
That doesnt help with CGNAT.
UPnP is just automating the process of forwarding ports, CGNAT will still screw you sideways because you're behind a router can't access or order around.
I don't know if bittorrent has improved - but 20 years ago I had a personal issue with it.
At that time our son was using it for games. He goes away to college and came home for the first school break. I get a phone call from our internet provider asking if our son was home. I was so shocked and handed the phone to our son.
Apparently at that time bittorrent was optimizing for the most efficient path to a host. Since we had relatively good connection, the mighty weight of the internet was funnelling through our tiny internet provider to our son's computer. The provider (without our knowing it) had made a deal with our son that he would only turn on bittorrent between midnight and 6 AM. I doubt other providers would be so generous.
I have been sceptical of bittorrent since that day.
All clients today (and probably back then) have options to limit bandwidth consumption including throttling, scheduling, and total data transfer caps. For serving mostly HTML and images, dedicating even 10% of a home broadband connection to serving content would allow many, many people per day to access archived pages.
[dead]
When the internet archive censors a website is it deleted permanently or just not publicly available?
I don't think they censor anything, strictly archiving. Do you know of any instance in which they censored a site?
http://web.archive.org/web/20240000000000*/twitter.com/taylo...
For one. I'm just curious what their policy is.
[delayed]
Kiwi farms
They black out all items that get a DMCA complaint or similar request (so it's still there just not accessible). However they permanently delete illegal stuff.
I would assume they delete illegal stuff as they are compelled to. What I'd like to know is their policy for legal stuff that they exclude that is not as a result of DMCA.
Can you give an example?
EDIT: Just seen your other reply. Perhaps it was excluded due to right to forget laws?