Home | Blog | Software | Reviews and Features | Forum | Help | Donate | About us
topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • September 25, 2016, 07:20:08 AM
  • Proudly celebrating 10 years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Please help Archive Team save Friendfeed's content  (Read 4476 times)

app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,650
    • View Profile
    • App's Apps
    • Read more about this member.
    • Donate to Member
Please help Archive Team save Friendfeed's content
« on: April 06, 2015, 11:05:30 AM »
Facebook is shutting Friendfeed down on April 9, 2015.

Archive Team is attempting to grab as much of the public content as possible, before the shutdown removes it from the internet, for good. All content grabbed will eventually be posted to archive.org's Wayback Machine.

These are the same guys that saved most of Geocities, when Yahoo shut it down. (they have also rescued content from other sites)

As an active, dedicated Friendfeed user, I have a personal interest in the success of this project. I have been in contact with them, answering some of their questions about Friendfeed, providing them with other info (api limitations, etc), and doing some community relations work related to this enormous project. (like making this post)

What they need now is at least 1000 volunteers to run a Warrior for this project, right now, or they will have no hope of being able to grab all of the public content in time, before the shut down.

If you qualify to run a Warrior, and you care about what they are trying to do here, please run one for at least this project.

To qualify, Please ensure the following:

  • No OpenDNS. No ISP DNS that redirects to a search page. Use non-captive DNS servers.
  • No ISP connections that inject advertisements into web pages.
  • No proxies. Proxies can return bad data.
  • No content-filtering firewalls.
  • No censorship. If you believe your country implements censorship, do not run a warrior.
  • No Tor. The original IP address is needed for the WARC file.
  • No free wifi cafe. Archiving your cafe's wifi service agreement repeatedly is not helpful.
  • We prefer connections from many public IP addresses if possible. (For example, if your apartment building uses a single IP address, we don't want your apartment banned.)


The Warrior runs on Windows, OS X and Linux using a virtual machine. You'll need one of the following:

  • VirtualBox (recommended)
  • VMware workstation/player


Or alternatively, if you are running Linux, you can try running a script for this project, directly: https://github.com/A...Team/friendfeed-grab

For more info about running a Warrior for this project, and other related project info, please see the following page: http://archiveteam.o...endFeed#Content_grab

For more general info about running a Warrior for this or any other Archive Team project, please see this page: http://archiveteam.o...ex.php?title=Warrior

Archiveteam[1].jpg

PS: DonationCoder has had a Friendfeed page for quite a long time, and this project will rescue it's posts, likes, subscriber list, subscriber comments, etc, to save them for future generations, too.



« Last Edit: April 06, 2015, 11:16:05 AM by app103 »

TaoPhoenix

  • Supporting Member
  • Joined in 2011
  • **
  • Posts: 4,548
    • View Profile
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #1 on: April 06, 2015, 11:15:06 AM »

I doubt I'm in the market for this, but I support the effort against yet one more "launch and dump" trend by the big corps!

>:(

app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,650
    • View Profile
    • App's Apps
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #2 on: April 06, 2015, 11:20:10 AM »
I doubt I'm in the market for this, but I support the effort against yet one more "launch and dump" trend by the big corps!

>:(

At this point, there is no hope that anyone can change Facebook's mind and stop the shut down, but archiving the public content to archive.org can still be accomplished, if enough people are willing to pitch in and help with that enormous task.

TaoPhoenix

  • Supporting Member
  • Joined in 2011
  • **
  • Posts: 4,548
    • View Profile
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #3 on: April 06, 2015, 11:24:16 AM »
I doubt I'm in the market for this, but I support the effort against yet one more "launch and dump" trend by the big corps!

>:(

At this point, there is no hope that anyone can change Facebook's mind and stop the shut down, but archiving the public content to archive.org can still be accomplished, if enough people are willing to pitch in and help with that enormous task.

Right, though do please provide guidance if people on generic home connections like Time Warner can help. Not counting a super weird blip last night, it "seems okay, but very erratic", but more importantly recalling some old legal verbiage from Verizon, they used to have rules against "servers" on home connections and I have no idea what the current status of Time Warner is.


app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,650
    • View Profile
    • App's Apps
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #4 on: April 06, 2015, 11:48:48 AM »
they used to have rules against "servers" on home connections

You would not be running a server. Nobody will be connecting to you. You will be running a grabber that will grab chunks of content off the friendfeed site and then upload it to Archive Team's servers. As long as you have a connection that meets the qualifications spelled out, you can do this on a home internet connection.


I, personally, can not provide any support for running a Warrior, but if anyone runs into any issues, I can ask a member of Archive Team to reply to anyone that needs some help getting it running.

TaoPhoenix

  • Supporting Member
  • Joined in 2011
  • **
  • Posts: 4,548
    • View Profile
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #5 on: April 06, 2015, 12:39:23 PM »
they used to have rules against "servers" on home connections

You would not be running a server. Nobody will be connecting to you. You will be running a grabber that will grab chunks of content off the friendfeed site and then upload it to Archive Team's servers. As long as you have a connection that meets the qualifications spelled out, you can do this on a home internet connection.


I, personally, can not provide any support for running a Warrior, but if anyone runs into any issues, I can ask a member of Archive Team to reply to anyone that needs some help getting it running.

Hmm.
Well, please post an intro post with some basic instructions for us and I will try to test if I can run one of these warrior things!


mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 36,247
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #6 on: April 06, 2015, 12:53:21 PM »
The Archive Team project looks pretty cool.  :up:

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 7,640
    • View Profile
    • The Blog of Deozaan
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #7 on: April 06, 2015, 01:12:39 PM »
Though I am looking into setting up a "Warrior" myself, DC probably isn't the best place to find 1,000 volunteers. You might get better visibility for this using reddit or slashdot or something.


app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,650
    • View Profile
    • App's Apps
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #8 on: April 06, 2015, 01:33:01 PM »
Though I am looking into setting up a "Warrior" myself, DC probably isn't the best place to find 1,000 volunteers. You might get better visibility for this using reddit or slashdot or something.

I didn't expect to find all 1000 qualified volunteers here, and there are posts elsewhere, including on Friendfeed, G+, and Frenf.it, and I believe that a member of Archive Team has made a post on Reddit (or is going to).

The Archive Team project looks pretty cool.  :up:

If you have the time to spare, you should see the action from inside their IRC channel on efnet #humancentifeed It's quite an interesting show, just as it was when they were grabbing Geocities.  ;)

Hmm.
Well, please post an intro post with some basic instructions for us and I will try to test if I can run one of these warrior things!

Original post above has links to instructions about setting up and running a Warrior. Try following those instructions, first, and if you have any questions or problems, ask them here and I'll try to get someone to answer them.

TaoPhoenix

  • Supporting Member
  • Joined in 2011
  • **
  • Posts: 4,548
    • View Profile
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #9 on: April 06, 2015, 01:36:44 PM »

" Basic usage
The warrior runs on Windows, OS X and Linux using a virtual machine. You'll need one of:

    VirtualBox (recommended)
    VMware workstation/player (free-gratis for personal use)
    See below for alternative virtual machines "

Sorry then, much as I like the spirit, I'll plead "too flighty" to get involved. I've never used a virtual machine before and I just don't have enough other interest to learn it for this!
:(

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 7,640
    • View Profile
    • The Blog of Deozaan
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #10 on: April 06, 2015, 06:45:45 PM »
It's pretty easy to set up. Just install virtualbox (or use the portable edition) and then follow the other few steps they have there.


4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,394
    • View Profile
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #11 on: April 06, 2015, 11:17:01 PM »
If it didn't require the use of a VM it would be a good use of all the unused RasPi's sitting around doing nothing.

tomos

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 10,238
    • View Profile
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #12 on: April 07, 2015, 03:07:24 AM »
It's pretty easy to set up. Just install virtualbox (or use the portable edition) and then follow the other few steps they have there.

that prompted me to give it a go - do ^that^ and
then this:
http://archiveteam.o...ex.php?title=Warrior

in fairness it does look very easy - and if I say that, it's *got* to be easy :D
When I read VM initially, I was thinking OS install etc. etc. but they seem to have all that covered :up:
Tom

tomos

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 10,238
    • View Profile
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #13 on: April 07, 2015, 04:21:54 AM »
it's worth noting that VirtualBox will not work if you have a modified version of uxtheme.dll (which has to be modified if you want to use non-windows themes for the OS).
https://www.virtualbox.org/ticket/13767

Should work otherwise (and/or if you replace/restore the dll).
Tom

app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,650
    • View Profile
    • App's Apps
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #14 on: April 07, 2015, 08:26:26 AM »
If it didn't require the use of a VM it would be a good use of all the unused RasPi's sitting around doing nothing.

There is a way to do it without the VM, running the script directly, on Linux, but it's a bit more complicated: http://archiveteam.o..._the_script_manually

app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,650
    • View Profile
    • App's Apps
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #15 on: April 07, 2015, 03:15:46 PM »
For anyone interested, you can watch the rescue in real-time, here: http://tracker.archi...team.org/friendfeed/

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 7,640
    • View Profile
    • The Blog of Deozaan
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #16 on: April 07, 2015, 07:29:18 PM »
For anyone interested, you can watch the rescue in real-time, here: http://tracker.archi...team.org/friendfeed/

No wonder I can't do anything! Kenshin is doing like a bajillion items per minute! I just keep getting messages saying things like:

"Tracker rate limiting is active. We don't want to overload the site we're archiving, so we've limited the number of downloads per minute. Retrying after 300 seconds..."

Impressive!

P.S. The first two items I did took over 10 hours each. Everything else has gone much faster. I guess I just got a couple of accounts who posted a bajillion things on their friendfeeds.


Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 7,640
    • View Profile
    • The Blog of Deozaan
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #17 on: April 08, 2015, 03:05:27 AM »
Holy carp. It's virtually finished!

There are no more items on the to-do list. And there are only about 1,500 items checked out that are awaiting finalization. I suspect if they time out, Kenshin will nab them up very quickly and take care of it. Kenshin did over 70% of the total archiving!

I guess I can shut my Warrior down. There's nothing left for me to do.

:Thmbsup:


f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,027
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #18 on: April 08, 2015, 03:26:03 AM »
it's worth noting that VirtualBox will not work if you have a modified version of uxtheme.dll (which has to be modified if you want to use non-windows themes for the OS).
https://www.virtualbox.org/ticket/13767
Damn, that's arrogant!
- carpe noctem

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 8,266
  • "In my dreams, I always do it right."
    • View Profile
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #19 on: April 08, 2015, 10:27:26 AM »
it's worth noting that VirtualBox will not work if you have a modified version of uxtheme.dll (which has to be modified if you want to use non-windows themes for the OS).
https://www.virtualbox.org/ticket/13767
Damn, that's arrogant!

That was my first thought, also- though I'd assume it would be possible to fork it and remove that check.

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,261
    • View Profile
    • www.StoicJoker.com
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #20 on: April 08, 2015, 11:29:10 AM »
it's worth noting that VirtualBox will not work if you have a modified version of uxtheme.dll (which has to be modified if you want to use non-windows themes for the OS).
https://www.virtualbox.org/ticket/13767
Damn, that's arrogant!

That was my first thought, also- though I'd assume it would be possible to fork it and remove that check.

Not everything UI plays nice together. T-Clock had a similar check in its code for many years. For the most part I'd just left it there - trusting the previous author - to see if anybody ever complained ... which they never did. TBH I'm not entirely sure if it's still there of not.

app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,650
    • View Profile
    • App's Apps
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #21 on: April 08, 2015, 07:19:43 PM »
Screenshot - 4_8_2015 , 6_14_01 PM.png


f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,027
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: Please help Archive Team save Friendfeed's content
« Reply #22 on: April 09, 2015, 03:22:30 AM »
 :Thmbsup:
- carpe noctem