YaCy - Decentralised Search

[Renegade]

I just found out about this:

http://yacy.net/en/

Web Search by the people, for the people

YaCy is a free search engine that anyone can use to build a search portal for their intranet or to help search the public internet. When contributing to the world-wide peer network, the scale of YaCy is limited only by the number of users in the world and can index billions of web pages. It is fully decentralized, all users of the search engine network are equal, the network does not store user search requests and it is not possible for anyone to censor the content of the shared index. We want to achieve freedom of information through a free, distributed web search which is powered by the world's users.

[attachimg=#]

Via: https://www.techdirt.com/articles/20140701/03143327738/distributed-search-engines-why-we-need-them-post-snowden-world.shtml

Rather timely as I've been finding StartPage doesn't index a lot of things (because it's basically just Google results), so I often end up using DuckDuckGo, which works beautifully. Still, another option besides DDG is welcome.

But, this could be really, really big. If this (or something similar) can effectively compete with Google, we well may see a world where Google actually needs to seriously fight off the censors instead of capitulating so much as they do now. (I've noticed quite a bit of censorship in Google over the recent past - but I am lacking the smoking gun proof...)

[IainB]

Thanks for posting about this. I just read that TechDirt post in my Bazqux feedreader and searched the DCF for references and found that you had just started this discussion thread.
Very thought-provoking.

[wraith808]

I just found out about this:

http://yacy.net/en/

Interesting- I'll have to keep an eye on this.  Have you installed it yet?

(re Smoking Gun Proof, check google vs bing vs DDG.  That was enough of a smoking gun for me)

[Renegade]

Have you installed it yet?

In a word...

AAARRRGGGHHH!!!

No.

My SSD is near full and I don't dare install anything more. I think I'm going to have to get a much larger SSD and clone the drive. I've been cleaning it out forever now, and just can't do much more.

However, this is a very high priority once I have enough space.

(re Smoking Gun Proof, check google vs bing vs DDG.  That was enough of a smoking gun for me)

If I may be so bold, I believe that Google is actively "shaping" search results for ideological purposes. I've done some searches that turn up zero in Google (and it's sidekick, StartPage), but are right at the top of DuckDuckGo. I can't possibly imagine Google being that incompetent that they "missed" some of the things I've tried to look into. I noticed this quite a while back, but set it aside as a possible "quirk". However, having done enough searches on a variety of topics, I can't push that suspicion aside any longer. Google is not a neutral search engine. And neutrality is the most important thing for a search.

[Renegade]

Here's one (bad) example... "michael meekin alien hat" (I had originally misspelled the name). Put that in StartPage or Google, then check DuckDuckGo. The site you are looking for is this: http://www.stopabductions.com/

(I like UFO videos/documentaries as they're often better than feature length films. I searched for that on a lark.)

BTW - The site there is just WILD! It's quite literally wilder than most good movie plots! 

DDG was better & found what I was looking for despite the spelling error. Google is pretty sophisticated, so I'm not quite sure what to make of it. Adding in "alien hat" should get to the proper resource.

But, that's the only example that I can remember at the moment. There have been others where DDG proved better. I should start taking notes...

[Renegade]

Interesting- I'll have to keep an eye on this.  Have you installed it yet?

I upgraded my drive to a 1 TB SSD and just installed YaCy - seems ok so far. It's not going to replace your regular search engine right now, but it is very promising.

You can browse it at http://localhost:8090/

I find I'm getting a lot of German results, but, it's still small, so I guess I'll run a node for now & hope for more people to jump on board. It will improve 1 person/installation at a time.

In a quick test, I started crawling my blog site, then searched for the title of an essay that I mirrored on in after giving the crawler some time. My blog post with the mirrored essay came up, but the original essay didn't. No real surprise there. But it shows that crawled searches from your local crawl come up darn fast.

The administration interface is very good as well. Lots in there. I've not explored it all yet, but it seems quite robust from a quick browse.

The quality of results is not at the DuckDuckGo or Google level yet, so that's a problem. I suppose that what YaCy needs is philosophically aligned people to start running nodes.

There are some interesting things that you could do with it though. For example, you could run through TOR, crawl sites that you are interested in, e.g. https://publicintelligence.net/ (a subversive site - they even host "Inspire" magazine!), then later search your own local cache in stealth mode while not connected through TOR. Since you'd have that locally stored, you'd never access the Internet (stealth mode) and could get search results without anyone knowing -- it's all done on your local machine.

tl;dr - Great software. Network isn't prime time ready yet. Needs more privacy advocates to add nodes for network effect to start to take hold.

[Renegade]

Seems ComputerWorld sees a need for something else...

http://www.computerworld.com/s/article/9249675/Why_we_need_an_underground_Google

Why we need an underground Google
Governments are forcing search engines to show wrong results. It's time for search engines to go rogue so they can be right.

More at the link.

[Deozaan]

It sounds interesting, but I guess I don't really get how it works. And can't you get in trouble for crawling sites? I recall reading about a guy who crawled the (entire?) internet and later got in trouble for it. I can't remember the details though. . .

[intlmedics]

Hi Guys,

We are currently conducting our own desktop serverless programs, for email, chat, messenger, www search, etc etc etc, and I read about Yacy and others whom just basically index the search with already made SE's out their, what we are conducting is part serverless part node, part server with built in 256 encryption for the uers search engine built in.

Nothing is stored within user Desktop GUI when they perform the www searches, and seeing we own the actual search node, we can then start marketing and add in all users and company www references/indexing, same as they all do.

Our phase 1 launch date is about 2 months from now.

Yes everything is FREE for the complete programs for life incl all updates and other apps. As we will then be building the serverless nodes for any on-line business that requires privacy for them and their users/members.