Main Area and Open Discussion > Living Room

What are the rules for local v. public index/search of DC Forum + Wayback saves?

<< < (2/3) > >>

mouser:
and i should note that google tends to index dc forum posts VERY quickly -- usually within the hour.

--- End quote ---

at least they used to -- i did some searches for posts from today and can't find them.

why isn't google indexing our pages quickly now.. i have no idea. is there a way to find out? doesn't appear so. is there someone to ask? doesn't appear so.  what do we do? nothing. just wait for the gods of google to decide who they want to favor or disfavor as usual.

IainB:
Thanks for the info about the rules @mouser.
...we welcome the wayback machine to index everything
_____________________
-mouser (October 05, 2015, 06:04 PM)
--- End quote ---
- presumably that refers to "everything" except that which is blocked by robots.txt or other means?

Interesting about the public search content indexing latency, but which failed google/duckgo search is temporary and due to this latency, and which is permanent and due to some content never being indexed? Could the latency or failure to index be attributable to long server response times to the crawlers, or something, which could lead to default timeouts if maximum response time performance thresholds were exceeded?
I often find the DCF website to be sluggish, but had attributed that to FF not the website.

None of that accounts for this sort of - what seems to be - date/time error though:



I could make a guess as to how it got that date/time, but it still seems wrong.
___________________________________

f0dder:
2. The crawlers of different search engines work differently, and to varying success. Unfortunately, I have to report I like DuckGo's philosophy, but that its engine simply gave me mediocre results a lot like you are saying, you can be staring at the relevant DC post in one window and the DuckGo simply refuses to pull it.

Then change engines and as much as I *don't* like their philosophy, Google IS basically the best except weird cases. I've done the same search in like 4 engines (including Binged-Yahoo and Startpage) and they come up with different results of varying quality. So it can't be a block per se if Google finds it and Duck doesn't.-TaoPhoenix (October 05, 2015, 04:35 PM)
--- End quote ---
That's my experience as well. I tried using DuckDuckGo for a while after Google was forced to obey the atrocious "right to be forgotten" EU crap, but DDG had pathetic results :(

f0dder:
why isn't google indexing our pages quickly now.. i have no idea. is there a way to find out? doesn't appear so. is there someone to ask? doesn't appear so.  what do we do? nothing. just wait for the gods of google to decide who they want to favor or disfavor as usual.-mouser (October 05, 2015, 06:18 PM)
--- End quote ---
The Google Webmaster Tools say nothing?

mouser:
The Google Webmaster Tools say nothing?
--- End quote ---

no not really. they i did check the sitemap, and it is listing recent pages, and it does show as being parsed by google without errors.  google webmaster tools shows that they have "decided" not to index most of the pages in the sitemap, though it doesn't say why. welcome to google world.

Navigation

[0] Message Index

[#] Next page

[*] Previous page