topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • October 17, 2019, 11:02 PM
  • Proudly celebrating 13 years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: IDEA: Searchengine Scraper  (Read 981 times)

The Code Queryer

  • Participant
  • Joined in 2019
  • *
  • default avatar
  • Posts: 33
    • View Profile
    • Donate to Member
IDEA: Searchengine Scraper
« on: April 06, 2019, 02:00 PM »
Hi,

My idea is a searchengine scraper.
It is like this ....
You come to a webpage and you see a search box (like google and the like). You type a url and click the "Scrape SERPs" button.
Now, the web app would visit the SERP page and scrape all the result links. It would follow to the next SERP pages and do likewise until it has met the dept you put.
A spider that visits SERP pages and scrapes all the result links. It then saves them on the website's database under your member username. Others can search band see what you scraped by doing your Username search. Likewise you can do too.
The scraper would scrape not only the links but their anchor texts, page titles, page meta keywords and meta descriptions.
In other words, a searchengine scraper. A web app. Built with php.
Anybody can build this then do the community a favour by releasing the source code here and on the gpl so we can learn from your source code. I am php student. I reckon cURL is good for the job.

Anyone like this idea,. Give it a thumbs up! :up:

Just imagine, you can scrape any searchengine with this.
I have built a .exe one. Anyone who builds a .php one then I am willing to trade or willinbg to give you a copy if you give me the .php copy along with comments so I can learn from your code.

NetRunner

  • Participant
  • Joined in 2018
  • *
  • default avatar
  • Posts: 15
    • View Profile
    • Donate to Member
Re: IDEA: Searchengine Scraper
« Reply #1 on: April 12, 2019, 04:48 PM »
What for? Who is going to use this?

One can do something similar already with existing tools. My feed reader does that basically, except for grabbing every linked page, as excerpts are fine for me, but would be easy to make it grab the full pages.

nickodemos

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 70
    • View Profile
    • Donate to Member
Re: IDEA: Searchengine Scraper
« Reply #2 on: April 12, 2019, 05:34 PM »
I just cant see who would be willing to store all that kind of data for others to look at. I would imagine that it would be obsoleted fairly fast.

Simpler idea is find a way to post links you visited.