Main Area and Open Discussion > General Software Discussion

ScraperWiki

(1/1)

bob99:

I came across this site during my "travels" through one of the sites I visit and was talked about on a TWiT podcast.

ScraperWiki

From their about page:
What is ScraperWiki?

There’s lots of useful data on the internet – crime statistics, government spending, missing kittens…

But getting at it isn’t always easy. There's a table here, a report there, a few web pages, PDFs, spreadsheets… And it can be scattered over thousands of different places on the web, making it hard to see the whole picture and the story behind it. It’s like trying to build something from Lego when someone has hidden the bricks all over town and you have to find them before you can start building!

To get at data, programmers write bits of code called ‘screen scrapers’, which extract the useful bits so they can be reused in other apps, or rummaged through by journalists and researchers. But these bits of code tend to break, get thrown away or forgotten once they have been used, and so the data is lost again. Which is bad.

ScraperWiki is an online tool to make that process simpler and more collaborative. Anyone can write a screen scraper using the online editor. In the free version, the code and data are shared with the world. Because it's a wiki, other programmers can contribute to and improve the code.

If you're not a programmer, or if you want to tackle a large or complex task, you can ask the ScraperWiki team to organise it for you.

--- End quote ---

Searched through and didn't see it had been posted about before.  Not related to them, haven't used it or know of the pros/cons of something like this, but might be of interest.

JavaJones:
I haven't had a chance to look closely at this but the quoted description sounds quite promising, similar to something I was looking for a while back. Thanks for posting, I'll check it out!

- Oshyan

Navigation

[0] Message Index