Ubuntu QA:
BlogBrainstormPackage status
Log in
Ubuntu QA
The Ubuntu community has contributed 13716 ideas, 65290 comments, 1273844 votes

Idea #4959: BRAINSTORM: idea ranking model proposal



up
5
down
Written by gijsterbeek the 17 Mar 08 at 22:50. Category: Brainstorm.
Related to: Nothing/Others. Status: New
Description
The current scoring model is flawed. The only way to keep track of fresh ideas, is to wait while they roll by in the 'Last ideas' section. Popularity creates popularity. This generates polarization and bias to old ideas. I propose a better measurement and ranking system:

Each idea should be ranked against more dimensions than just popularity. Sure, appreciation is important, but so is controversy, clearness and time. Let's say we have a brainstorm idea X, which was posted 3 days ago and has 40 votes pro, 20 votes contra and 5 comments. We also have idea Y, 5 days old, which has 300 votes pro, 320 votes contra, and 100 comments. In the current situation, idea Y will be on top - forever. But we can derive several more useful indicators from these ideas.

Average appreciation (AA):

This is the current way of scoring, which is the amount of 'pro' voters minus the amount of 'contra' voters. This is how much the idea is liked by the brainstorm visitors. For idea X the AA would be +20, for idea Y it would be -20. This would indicate that action should be taken on idea X, rather than idea Y, because the nett amount of pro-votes was higher then the contra votes. This would however ignore the fact that idea Y has 300 pro votes, while idea X has left many people indifferent and generated only some 60 votes total. This brings us to another factor:

Controversy factor (CF):

Tells us something about the attention an idea gets from the BRAINSTORM visitors. Does the idea instantly 'trigger' the viewer? The amount of 'pro' voters plus the amount of 'contra' voters, divided by the absolute average appreciation. This would give idea X a CF of (40+20)/20 = 3. This is low compared to idea Y, which has a score of (300+320)/20, which results in 31. Now we see that idea Y has a higher visibility than idea X, even though the average appreciation of the idea is negative.

Comment-to-controversy factor (CC):

Tells us something about the completeness of an idea. If an idea is clear and evident, people will only vote. When information is missing, people will add comments. It would be wrong to confuse the comments with a discussion, because the current implementation of the comment system only allows for comments on the idea. Threaded comments (commenters commenting commenters) are not allowed. So the amount of all votes divided by the amount of comments would tell us something about how clearly scoped the idea is and what has to be done to implement it. Lower values mean sharper ideas.

Idea X would have a score of 3/5 = 0.6. Idea Y would have 31/100 = 0.31 Clearly, though the controversy about idea Y is way larger than the indifference surrounding idea X, it has only half of its CC. This makes sense. Obviously an idea with a clear description will create 'faster' opinions, therefore people will vote for idea Y without much hesitation, while idea X is just too confusing to vote for. Visitors have a short attention span. Which brings us to the next factor: time.

Gestation Time

On this site, new ideas will generate more votes on day 1 than they will do on any day after that. Ideas move to the background after 2 or 3 days, and will not be found back easily anymore. This is not how it should be. There should be two distict phases in the life of an idea (on BRAINSTORM). First, a 'gestation period'. The idea is new and needs a day or two to generate enough controversy AND/OR appreciation AND/OR comments per vote to be 'important'. The combined score on these 3 indicators will give it an 'importance' ranking, which is tied to more than just popularity. This will bring intriguing ideas to the top, while boring/unimportant ideas will stay low. As time marches on, ideas will lose importance (or better still: be implemented) and drop to lower ranks. Idea X, being 3 days old and having had 60 votes in total, would have 20 votes per day, while idea Y had 620/5=124 votes per day. Again, idea Y has a higher score then idea X, and therefore should be granted a place with 'the headlines'.

For an even better model, and taking half-life into consideration, only the votes and comments of the last 2 days or so should be taken into consideration. This prevents old ideas to remain on top.

Implementing something like this model would vastly increase the usability of this brainstorm, as well as encourage a lot of people to submit their ideas. Also, it would greatly decrease the amount of (relevant) duplicates, because people will only see relevant ideas. This means they don't need to search, which means they will just think of another idea to submit.

Also, ideas should be shorter ;)
Tags: (none)

Attachments
No attachments.


Duplicates


Comments
forteller wrote on the 18 Mar 08 at 01:34
Hey, I think you've got some good points and ideas here. These kinds of things definitely needs to be improved somehow here on Brainstorm! This deserves to be thought more about and discussed. I hope the admins will weigh in with their thought and will take this idea into consideration.

spyyder wrote on the 18 Mar 08 at 02:58
Clearly people are modding this down without reading, some good thoughts.

AdminAdmin nand (Brainstorm admin) wrote on the 18 Mar 08 at 10:56
Very interesting indeed!
Some nice suggestions that will help me on the looong and hard way to find the best "popular" ordering algorithm.

guyminuslife wrote on the 25 Mar 08 at 18:56
I disagree with a few points (largely, that a lack of comments means a sharper idea), but +1 for other factors.

jhoger wrote on the 25 May 08 at 20:49
Whether you want to admit it or not, there are "conversations" that take place in the comments. There really does need to be threads/moderation but given that it's not there, posters just refer to the other commenters by name or quote them.

FWIW, I often post a comment on very interesting ideas, because it usually triggers further riffs on the same idea or if there are minor issues I might mention them. Also, I often indicate why I voted for an idea, whether positive or negative or +0.

tchalvakspam wrote on the 12 Jul 08 at 15:03
"For an even better model, and taking half-life into consideration, only the votes and comments of the last 2 days or so should be taken into consideration. This prevents old ideas to remain on top."


But there's no such thing as an "old" idea if:
A: it's not yet implemented.
B: it's still gathering votes.
C: it's still gathering comments.

Maybe I'm just reading that wrong, because the rest of it did seem like a good in-depth analysis, though perhaps a bit too complicated to expect adoption? Simpler execution = better change of adoption.

On Comment-to-Controversy. I think that you're missing the point in your analysis that more comments mean a less clear idea. A vote takes half a second, a single click. A comment takes much more time, which means that people find the idea more important. It also means that an idea is more likely to have been read and analyzed well, as opposed to just scanned and voted on. More comments = a better analysis of the idea, for OR against.

So an idea with 20 + votes and 20 comments is better than an idea with just 20 + comments, by a large factor.
By the same token, an idea with 20 - votes and 20 comments is worse than an idea with just 20 - votes and no comments, because it implies extensive analysis of that first idea has occurred, and even after the analysis the idea was found lacking.

More comments should increase the importance of the vote tally, up or down, not just mark down an idea.


Post your comment