Author Topic: Netflix contest: $1M to developer of best movie recommendation engine (Read 41,987 times)

KenR · « **on:** October 02, 2006, 06:13 PM »

Netflix is holding a 1 million dollar contest awarded to the coder who can betst improve their movie recommendation engine.

http://www.netflixprize.com/rules

from http://oreilly.com/

Gerome · « **Reply #1 on:** October 02, 2006, 06:43 PM »

Hi,

Bah... I'd prefer 1 million Euros

KenR · « **Reply #2 on:** October 02, 2006, 06:47 PM »

Ok Gerome,
You want to try to convince us that is NOT complaining?
Ken

brotherS · « **Reply #3 on:** October 02, 2006, 07:06 PM »

Quote from: KenR on October 02, 2006, 06:13 PM

Netflix is holding a 1 million dollar contest awarded to the coder who can betst improve their movie recommendation engine.

What a great idea! I read articles in the past that Netflix seems to be innovative (I'm no customer nor related to them) and this proves it. By running the contest until "at least October 2, 2011" they show they really plan to spend the million to gain a 10% performance win, and I also like the idea of their "Progress Prizes: $50,000 (USD) Cash each award", which is handed out once a year for those who are improving the performance the most, but not reaching the 10% hurdle.

@Gerome: we'd still accept you if you'd show less negativity, really!

Gerome · « **Reply #4 on:** October 02, 2006, 07:10 PM »

Hi,

Quote from: brotherS on October 02, 2006, 07:06 PM

@Gerome: we'd still accept you if you'd show less negativity, really!

What about negativity ?
Because 1 million of Euro is > to 1 million of USD ?
ROFL

allen · « **Reply #5 on:** October 03, 2006, 04:01 AM »

Quote from: Gerome on October 02, 2006, 07:10 PM

Hi,

Quote from: brotherS on October 02, 2006, 07:06 PM
@Gerome: we'd still accept you if you'd show less negativity, really!

What about negativity ?
Because 1 million of Euro is > to 1 million of USD ?
ROFL

But perhaps the point, Gerome, is that 1 millsion USD > Nothing
-- and as an American company, USD would make sense as the prize, no?

Gerome · « **Reply #6 on:** October 03, 2006, 07:31 AM »

Hello,

Quote from: allen on October 03, 2006, 04:01 AM

But perhaps the point, Gerome, is that 1 millsion USD > Nothing
-- and as an American company, USD would make sense as the prize, no?

Ok Allen et al, I've just seen my jokes are just badly interpreted... I'm sorry for you...

BTW, this offer seems nice.

brotherS · « **Reply #7 on:** October 03, 2006, 08:02 AM »

Quote from: Gerome on October 03, 2006, 07:31 AM

I'm sorry for you...

And here you go again...

mouser · « **Reply #8 on:** October 03, 2006, 09:13 PM »

interesting blog critisizing the approach:
http://www.feedblog.org/2006/10/netflix_ranking.html

Quote

If you assume they can get 4x compression out of this dataset you're still looking at 50G which you'll probably have to query at runtime. This probably means a memory based cluster and to do that you'll need at least $15-20k at a minimum to get started.

What Netflix should do is allow teams to pitch them their proposals and fund their research with a winning prize of $1M.

JavaJones · « **Reply #9 on:** October 07, 2006, 01:10 AM »

I'm not a coder but it seems to me you wouldn't necessarily need to use the whole data set to prototype and see if your results net reasonable gains. I presume they are supporting this to a reasonable degree by providing a good amount of info about their current system (if not source code). If that's the case it should be reasonable to extrapolate results for your test data set size/range and work against that. If you can show significant potential in a new approach I would think they will work with you to scale it up and see if it's really practical. It only makes sense from their end to do so.

- Oshyan

mouser · « **Reply #10 on:** April 19, 2007, 09:42 PM »

Alex (www.3form.org) has convinced me to join with him in working on this a little since both of us have backgrounds in machine learning.

I have my doubts about the possibilities of winning this but it seems an interesting challenge and worth a little bit of time playing with it. I'm happy to talk with anyone in the donationcoder irc channel (#donationcoder on efnet, or hit the chat button aboce) who thinks they might also want to try entering.

urlwolf · « **Reply #11 on:** April 20, 2007, 11:34 AM »

all,

Do you know of any webservice where users log which movies they have actually seen? Something like last.fm for music, but using movies instead...

Actually, same thing for books read would be great too..
@mouser: I sent you an email, cannot reach the #IRC channel for some reason...

CWuestefeld · « **Reply #12 on:** April 20, 2007, 04:34 PM »

I made a stab at this late last year with a friend. The amount of data isn't a big deal for the storage capacity of a modern PC. More difficult is being able to process it in a reasonable time.

We started with a bit of success. Our very first attempt was enough to beat Netflix's baseline and yield a score that qualified for the leaderboard (at the time). Doing this was surprisingly easy. Nothing but a good working knowledge of statistics was needed to beat their baseline. Simply normalizing the scores of all the users according to their standard deviations was enough to do this.

Our project fizzled out after just a couple of weeks. The problem was that neither of us has an understanding of the collaborative filtering techniques that appear to be necessary to do really well. And experimentation was difficult. The size of the data set is such that we could only really make one run a day (we were using MS SQL Server, if you're wondering).

Also, I was a little frustrated because I think that their criteria measure the wrong thing. At first it was an intellectual challenge anyway, but that wore out. The challenge is to predict the score that a customer would assign for each of a set of movies. The thing is, this isn't really an interesting question to solve. I don't care very much if Netflix thinks that I'd give this movie a 3.1 or 3.5. What's really interesting is only selecting a list of movies at the top end of the scale, so Netflix can give me a list of recommendations when I ask "what good movies have you got for me today?".

mouser · « **Reply #13 on:** April 20, 2007, 06:06 PM »

great to hear you worked on it!!

my main source of skepticism is a suspicion/fear that there is some inherent noise in the data, and that winning the contest may in fact require being better than the noise would allow without pure luck. in other words, a person's rating for any given movie may vary a little bit depending on their mood at the time, or based on some truly unpredictable factors. for example, imagine i ask you to predict the votes of 100 people and i happen to know that 95 of them always vote one way, but the other 5 flip a coin to decide which way to vote. now you can't hope to get a perfect score because of the element of true randomness (coin flip). At some point, trying to get better than a certain score on the netflix challenge data is going to look like that -- though whether that happens well before the million dollar prize accuracy i dont know, but that's my fear.

mouser · « **Reply #14 on:** April 20, 2007, 06:07 PM »

the forum makes for great reading: http://www.netflixprize.com/community/

mouser · « **Reply #15 on:** September 13, 2007, 12:58 AM »

about 20 days until first round closes..
i'm playing around with some code and hope to have it done in time to submit.

tinjaw · « **Reply #16 on:** September 13, 2007, 01:01 AM »

Go mouser. Give us an update on your scores. I remember you talking about an ongoing scoring mechanism during the contest.

mouser · « **Reply #17 on:** June 27, 2009, 03:16 AM »

After 2 years it looks like there is going to be a grand winner.. amazing.

http://tech.slashdot.org/story/09/06/26/234247/Netflix-Prize-May-Have-Been-Achieved

The nice thing will be reading about the final algorithms.. we've actually known most of the details of it since the authors are academics and published papers on their algorithms.. But it will be interesting to hear how they squeezed out the last drops of performance.

mouser · « **Reply #18 on:** July 27, 2009, 06:15 AM »

The contest is now closed and the winner chosen, and WOW it seems like a surprising ending to a surprising contest..

By A Nose: Netflix Prize Leaders One-Upped With One Day Remaining:

Homepage of the winning group:

http://www.the-ensemble.com/

wow. just wow.

TheQwerty · « **Reply #19 on:** July 27, 2009, 10:58 AM »

Note that no winner has been chosen yet.
Netflix is going to evaluate the entries and also take performance into consideration before they select one or re-open the contest.

mouser · « **Reply #20 on:** July 27, 2009, 11:01 AM »

my understanding is that they were only supposed to verify that the code works, not decide based on efficiency, etc. and that they could not "re-open" the contest unless there was some violation of the rules.

do you have any more info about this and links?

mouser · « **Reply #21 on:** July 27, 2009, 11:02 AM »

I've read some of the previous papers published by both teams that broke the 10% barrier and it will be fascinating to read the final papers.

TheQwerty · « **Reply #22 on:** July 27, 2009, 12:03 PM »

Quote from: mouser on July 27, 2009, 11:01 AM

my understanding is that they were only supposed to verify that the code works, not decide based on efficiency, etc. and that they could not "re-open" the contest unless there was some violation of the rules.

do you have any more info about this and links?

Well for links, just look at what you posted, none of them have an official statement that Ensemble won.

And the official Netflix blog says that they'll begin verifying: http://blog.netflix.com/2009/07/netflix-prize-competition-closes-but-no.html

Quote from: "http://www.netflixprize.com/closed"

Qualified entries will be evaluated as described in the Rules. We look forward to awarding the Grand Prize, which we expect to announce in a few weeks. However if a Grand Prize cannot be awarded because no submission can be verified by the judges, the Contest will reopen. We will make an announcement on the Forum after the Contest judges reach a decision.

So maybe not taking performance into consideration, but there's also ...

Quote from: "http://www.netflixprize.com/community/viewtopic.php?pid=9237#p9237"

Thanks. In fact, this is a very happy day for us - our team is top contender for winning the Grand Prize, as we have a better Test score than The Ensemble. (Probably this is the first post revealing this in the forum smile)

Which led me to believe that the leaderboards are not calculated on the full/exact data set that the winning algorithm will be tested on.

Don't really know.. but I wouldn't get ahead of yourself congratulating The Ensemble until Netflix has made an announcement. Certainly not with results as close as these.

mouser · « **Reply #23 on:** July 27, 2009, 09:56 PM »

Quote

Which led me to believe that the leaderboards are not calculated on the full/exact data set that the winning algorithm will be tested on.

ah that's an interesting point.. i'm trying to remember now what the rules said about that.
it does seem like with scores this close, it's a bit silly to say that one of these 2 teams one, and the other "lost."

mouser · « **Reply #24 on:** August 12, 2009, 02:36 PM »

updates:
http://www.google.com/hostednews/ap/article/ALeqM5hrJlkUWtxCOQZJLIyUQzLsWXaNLAD9A0TFMO0
http://www.netflixprize.com/community/viewtopic.php?id=1520