Please help Archive Team save Friendfeed's content

[app103]

Facebook is shutting Friendfeed down on April 9, 2015.

Archive Team is attempting to grab as much of the public content as possible, before the shutdown removes it from the internet, for good. All content grabbed will eventually be posted to archive.org's Wayback Machine.

These are the same guys that saved most of Geocities, when Yahoo shut it down. (they have also rescued content from other sites)

As an active, dedicated Friendfeed user, I have a personal interest in the success of this project. I have been in contact with them, answering some of their questions about Friendfeed, providing them with other info (api limitations, etc), and doing some community relations work related to this enormous project. (like making this post)

What they need now is at least 1000 volunteers to run a Warrior for this project, right now, or they will have no hope of being able to grab all of the public content in time, before the shut down.

If you qualify to run a Warrior, and you care about what they are trying to do here, please run one for at least this project.

To qualify, Please ensure the following:

• No OpenDNS. No ISP DNS that redirects to a search page. Use non-captive DNS servers.
• No ISP connections that inject advertisements into web pages.
• No proxies. Proxies can return bad data.
• No content-filtering firewalls.
• No censorship. If you believe your country implements censorship, do not run a warrior.
• No Tor. The original IP address is needed for the WARC file.
• No free wifi cafe. Archiving your cafe's wifi service agreement repeatedly is not helpful.
• We prefer connections from many public IP addresses if possible. (For example, if your apartment building uses a single IP address, we don't want your apartment banned.)

The Warrior runs on Windows, OS X and Linux using a virtual machine. You'll need one of the following:

• VirtualBox (recommended)
• VMware workstation/player

Or alternatively, if you are running Linux, you can try running a script for this project, directly: https://github.com/A...Team/friendfeed-grab

For more info about running a Warrior for this project, and other related project info, please see the following page: http://archiveteam.o...endFeed#Content_grab

For more general info about running a Warrior for this or any other Archive Team project, please see this page: http://archiveteam.o...ex.php?title=Warrior

PS: DonationCoder has had a Friendfeed page for quite a long time, and this project will rescue it's posts, likes, subscriber list, subscriber comments, etc, to save them for future generations, too.

[TaoPhoenix]

I doubt I'm in the market for this, but I support the effort against yet one more "launch and dump" trend by the big corps!

[app103]

I doubt I'm in the market for this, but I support the effort against yet one more "launch and dump" trend by the big corps!

-TaoPhoenix (April 06, 2015, 11:15 AM)

At this point, there is no hope that anyone can change Facebook's mind and stop the shut down, but archiving the public content to archive.org can still be accomplished, if enough people are willing to pitch in and help with that enormous task.

[TaoPhoenix]

I doubt I'm in the market for this, but I support the effort against yet one more "launch and dump" trend by the big corps!

-TaoPhoenix (April 06, 2015, 11:15 AM)

At this point, there is no hope that anyone can change Facebook's mind and stop the shut down, but archiving the public content to archive.org can still be accomplished, if enough people are willing to pitch in and help with that enormous task.

-app103 (April 06, 2015, 11:20 AM)

Right, though do please provide guidance if people on generic home connections like Time Warner can help. Not counting a super weird blip last night, it "seems okay, but very erratic", but more importantly recalling some old legal verbiage from Verizon, they used to have rules against "servers" on home connections and I have no idea what the current status of Time Warner is.

[app103]

they used to have rules against "servers" on home connections

-TaoPhoenix (April 06, 2015, 11:24 AM)

You would not be running a server. Nobody will be connecting to you. You will be running a grabber that will grab chunks of content off the friendfeed site and then upload it to Archive Team's servers. As long as you have a connection that meets the qualifications spelled out, you can do this on a home internet connection.


I, personally, can not provide any support for running a Warrior, but if anyone runs into any issues, I can ask a member of Archive Team to reply to anyone that needs some help getting it running.

[TaoPhoenix]

they used to have rules against "servers" on home connections

-TaoPhoenix (April 06, 2015, 11:24 AM)

You would not be running a server. Nobody will be connecting to you. You will be running a grabber that will grab chunks of content off the friendfeed site and then upload it to Archive Team's servers. As long as you have a connection that meets the qualifications spelled out, you can do this on a home internet connection.


I, personally, can not provide any support for running a Warrior, but if anyone runs into any issues, I can ask a member of Archive Team to reply to anyone that needs some help getting it running.

-app103 (April 06, 2015, 11:48 AM)

Hmm.
Well, please post an intro post with some basic instructions for us and I will try to test if I can run one of these warrior things!

[mouser]

The Archive Team project looks pretty cool. 

[Deozaan]

Though I am looking into setting up a "Warrior" myself, DC probably isn't the best place to find 1,000 volunteers. You might get better visibility for this using reddit or slashdot or something.

[app103]

Though I am looking into setting up a "Warrior" myself, DC probably isn't the best place to find 1,000 volunteers. You might get better visibility for this using reddit or slashdot or something.

-Deozaan (April 06, 2015, 01:12 PM)

I didn't expect to find all 1000 qualified volunteers here, and there are posts elsewhere, including on Friendfeed, G+, and Frenf.it, and I believe that a member of Archive Team has made a post on Reddit (or is going to).

The Archive Team project looks pretty cool. 

-mouser (April 06, 2015, 12:53 PM)

If you have the time to spare, you should see the action from inside their IRC channel on efnet #humancentifeed It's quite an interesting show, just as it was when they were grabbing Geocities. 

Hmm.
Well, please post an intro post with some basic instructions for us and I will try to test if I can run one of these warrior things!

-TaoPhoenix (April 06, 2015, 12:39 PM)

Original post above has links to instructions about setting up and running a Warrior. Try following those instructions, first, and if you have any questions or problems, ask them here and I'll try to get someone to answer them.

[TaoPhoenix]

" Basic usage
The warrior runs on Windows, OS X and Linux using a virtual machine. You'll need one of:

    VirtualBox (recommended)
    VMware workstation/player (free-gratis for personal use)
    See below for alternative virtual machines "

Sorry then, much as I like the spirit, I'll plead "too flighty" to get involved. I've never used a virtual machine before and I just don't have enough other interest to learn it for this!

[Deozaan]

It's pretty easy to set up. Just install virtualbox (or use the portable edition) and then follow the other few steps they have there.

[4wd]

If it didn't require the use of a VM it would be a good use of all the unused RasPi's sitting around doing nothing.

[tomos]

It's pretty easy to set up. Just install virtualbox (or use the portable edition) and then follow the other few steps they have there.

-Deozaan (April 06, 2015, 06:45 PM)

that prompted me to give it a go - do ^that^ and
then this:
http://archiveteam.o...ex.php?title=Warrior

in fairness it does look very easy - and if I say that, it's *got* to be easy
When I read VM initially, I was thinking OS install etc. etc. but they seem to have all that covered

[tomos]

it's worth noting that VirtualBox will not work if you have a modified version of uxtheme.dll (which has to be modified if you want to use non-windows themes for the OS).
https://www.virtualbox.org/ticket/13767

Should work otherwise (and/or if you replace/restore the dll).

[app103]

If it didn't require the use of a VM it would be a good use of all the unused RasPi's sitting around doing nothing.

-4wd (April 06, 2015, 11:17 PM)

There is a way to do it without the VM, running the script directly, on Linux, but it's a bit more complicated: http://archiveteam.o..._the_script_manually

[app103]

For anyone interested, you can watch the rescue in real-time, here: http://tracker.archi...team.org/friendfeed/

[Deozaan]

For anyone interested, you can watch the rescue in real-time, here: http://tracker.archi...team.org/friendfeed/

-app103 (April 07, 2015, 03:15 PM)

No wonder I can't do anything! Kenshin is doing like a bajillion items per minute! I just keep getting messages saying things like:

"Tracker rate limiting is active. We don't want to overload the site we're archiving, so we've limited the number of downloads per minute. Retrying after 300 seconds..."

Impressive!

P.S. The first two items I did took over 10 hours each. Everything else has gone much faster. I guess I just got a couple of accounts who posted a bajillion things on their friendfeeds.

[Deozaan]

Holy carp. It's virtually finished!

There are no more items on the to-do list. And there are only about 1,500 items checked out that are awaiting finalization. I suspect if they time out, Kenshin will nab them up very quickly and take care of it. Kenshin did over 70% of the total archiving!

I guess I can shut my Warrior down. There's nothing left for me to do.

[f0dder]

it's worth noting that VirtualBox will not work if you have a modified version of uxtheme.dll (which has to be modified if you want to use non-windows themes for the OS).
https://www.virtualbox.org/ticket/13767

-tomos (April 07, 2015, 04:21 AM)

Damn, that's arrogant!

[wraith808]

it's worth noting that VirtualBox will not work if you have a modified version of uxtheme.dll (which has to be modified if you want to use non-windows themes for the OS).
https://www.virtualbox.org/ticket/13767

-tomos (April 07, 2015, 04:21 AM)

Damn, that's arrogant!

-f0dder (April 08, 2015, 03:26 AM)

That was my first thought, also- though I'd assume it would be possible to fork it and remove that check.

[Stoic Joker]

it's worth noting that VirtualBox will not work if you have a modified version of uxtheme.dll (which has to be modified if you want to use non-windows themes for the OS).
https://www.virtualbox.org/ticket/13767

-tomos (April 07, 2015, 04:21 AM)

Damn, that's arrogant!

-f0dder (April 08, 2015, 03:26 AM)

That was my first thought, also- though I'd assume it would be possible to fork it and remove that check.

-wraith808 (April 08, 2015, 10:27 AM)

Not everything UI plays nice together. T-Clock had a similar check in its code for many years. For the most part I'd just left it there - trusting the previous author - to see if anybody ever complained ... which they never did. TBH I'm not entirely sure if it's still there of not.

[app103]

[f0dder]