topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Thursday March 28, 2024, 4:36 pm
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Added delay to TSearch.py to make the script more human  (Read 10487 times)

wjamoe

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 99
    • View Profile
    • Donate to Member
Added delay to TSearch.py to make the script more human
« on: March 03, 2013, 08:43 AM »
My list contains 600 items, ms will block after about 30 results.


I added a time.sleep(60) command in TSearch.py (and had to import time) to fool the (b i n g) search engine.

Each input line takes 60 seconds, it makes the script more human.....

So I don't have to enter lost of  subselections (600=20x30 input items)  :D



See both changes in the code below between the line with ======== on it

Code: Python [Select]
  1. #-------------------------------------------------------------------------------
  2. # imports we need
  3. # include top level directory (needed to import from helper below)
  4. import sys
  5. import os
  6. #201302 wjamoe added import time ===================================
  7. import time
  8. #=============================================================

Code: Python [Select]
  1. def DoSearch(self,id,label,search):
  2.         # this function does the main work -- take a search term
  3.         # return a list of results
  4.         results = []
  5.  
  6.         #initialize result with engine name and id
  7.         baseresult = {"engine":self.get_searchenginename()}
  8.         if id!="":
  9.             baseresult["id"]=id
  10.         if label!="":
  11.             baseresult["label"]=label
  12.  
  13.         # ok grab a web page using our search terms
  14.         content = self.SearchWebPageGrabContents(search)
  15.  
  16.         # and now parse results
  17.         addresults = self.ParseWebPageContentIntoResults(search,content,baseresult)
  18.         #201302 wjamoe added delay of 60 seconds =============================
  19.         time.sleep(60)
  20.         #=============================================================
  21.         # add
  22.         results.extend(addresults)
  23.         # return
  24.         return results

The testrun for 2 items takes 120 seconds  :(

wiem

Ath

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 3,612
    • View Profile
    • Donate to Member
Re: Added delay to TSearch.py to make the script more human
« Reply #1 on: March 03, 2013, 09:06 AM »
Disclaimer: I don't know how to program in python (at all, yet)
Couldn't you keep start-time, fire 5 or 10 searches, then wait until 60 seconds have passed since start-time, and fire another set of searches? Would take much less waiting. But it would also look/feel less human-like searching...

wjamoe

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 99
    • View Profile
    • Donate to Member
Re: Added delay to TSearch.py to make the script more human
« Reply #2 on: March 03, 2013, 09:33 AM »
That would require to reshuffle code, and will take more time to program and a lot more testing to find out what the heuristics are within the search engines. The engines probably all differ in that respect to identify robots and alike.
Current solution suffices for me, sorry...:-[

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Added delay to TSearch.py to make the script more human
« Reply #3 on: March 03, 2013, 11:45 AM »
I'm just thrilled that someone other than me is using this tool.  :P