avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Friday September 30, 2022, 3:52 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: ScraperWiki  (Read 1933 times)


  • Supporting Member
  • Joined in 2008
  • **
  • default avatar
  • Posts: 345
    • View Profile
    • Donate to Member
« on: October 13, 2011, 01:01 PM »

I came across this site during my "travels" through one of the sites I visit and was talked about on a TWiT podcast.


From their about page:
What is ScraperWiki?

There’s lots of useful data on the internet – crime statistics, government spending, missing kittens…

But getting at it isn’t always easy. There's a table here, a report there, a few web pages, PDFs, spreadsheets… And it can be scattered over thousands of different places on the web, making it hard to see the whole picture and the story behind it. It’s like trying to build something from Lego when someone has hidden the bricks all over town and you have to find them before you can start building!

To get at data, programmers write bits of code called ‘screen scrapers’, which extract the useful bits so they can be reused in other apps, or rummaged through by journalists and researchers. But these bits of code tend to break, get thrown away or forgotten once they have been used, and so the data is lost again. Which is bad.

ScraperWiki is an online tool to make that process simpler and more collaborative. Anyone can write a screen scraper using the online editor. In the free version, the code and data are shared with the world. Because it's a wiki, other programmers can contribute to and improve the code.

If you're not a programmer, or if you want to tackle a large or complex task, you can ask the ScraperWiki team to organise it for you.

Searched through and didn't see it had been posted about before.  Not related to them, haven't used it or know of the pros/cons of something like this, but might be of interest.


  • Review 2.0 Designer
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 2,739
    • View Profile
    • Donate to Member
Re: ScraperWiki
« Reply #1 on: October 13, 2011, 03:40 PM »
I haven't had a chance to look closely at this but the quoted description sounds quite promising, similar to something I was looking for a while back. Thanks for posting, I'll check it out!

- Oshyan