Stepmania Leaderboards Released!

[Wafles]

Re: Stepmania Leaderboards Released!

Yeah that's def. part of it. It's also majorly boosting the difficulty of the trills at the end of the file. In short, the algorithm makes passes on the file for assumed player difficulty "X", increasing the value by 0.01 every time. Once it gets 93% that becomes the difficulty of the file.

What seems to be happening is that the trills at the end of the file just kill runs by causing a lot of misses because our algorithm sees them as jacks (and as such increases their difficulty). It's being overdone by a fair bit at the moment.

[portalbob340]

Re: Stepmania Leaderboards Released!

Is there any way to distinguish charts of different difficulties for a song?

[MinaciousGrace]

Re: Stepmania Leaderboards Released!

Ok before any witch hunt to disparage the algorithm gets out of hand I want to state a few things clearly:

If something is obviously wrong, then I already know about it.

Making tendentious conjectures about the operations and deficiencies of a system I created by myself is not helpful. Doing so necessarily takes the stance of presuming that you are more an expert on my own work than I am. If any of you wish to continue to do this I ask that you preface your remarks with "Given that I know more about the algorithm than its creator..."

There is a delay between resolving issues and testing of the adjustments using my local parser and Wafles updating the version on the site. Many of the current issues are already resolved and awaiting implementation.

The entire point of having an algorithm is accountability. The system has been designed from the ground up to facilitate detection and rectification of issues that result in erroneous scoring. If I can't resolve the issues to a satisfactory degree, then I am accountable.

I also must dispel any illusion that this tool is somehow collectively owned by the community. It is not. It is a tool that I created and thanks to Wafles have made available for the community to make use of. I gain nothing from the hundreds of hours of work I've spent on this project and I am not so naive as to expect any compensation or thanks in return.

What I do expect however, even as unreasonable as it is, is some modicum of acknowledgement that I am capable of recognizing glaring issues and resolving them on my own.

A decision was made to release the site for public use as early as possible with the clear indication that it is still a work in progress. This is very clearly stated on the site and so is virtually everything else in this post.

Aside from the well documented issues with the algorithm, the core functionalities of the site remain intact. If you have a genuine interest in helping me refine this tool the best possible thing you can do is submit scores and suggest songs and rates so that large sample sizes I cannot quickly reproduce on my own are available for analysis concerning any pernicious underlying trends.

Even if you have no interest in helping me, this is the entire point of the site, Unless your expectation is that no updates will ever be made to it your time is probably better served just doing that.

If you just want to sit here and shitmonger then I guess you are free to do so. If you want to shitmonger in whatever various skype groups you are free to do that as well, however doing so neither helps nor hurts my efforts, making it the most pointless course of action regardless of your intent.

@choof:
"
A file will not be ranked for any of the following reasons:
- Has not been and cannot be “AA”d.
- Is assigned a wildly incorrect difficulty due to some oversight in the scoring algorithm. These instances will be investigated and the underlying cause will be rooted out and corrected, and once done files removed or blocked for this reason will be ranked assuming they do not meet any other criteria for exclusion.
- Is a “vibro” file. In practice vibro files must be considered too differently from traditional files in terms of difficulty scoring as a pad file would be. Also nobody cares about vibro.
"

Otherwise files that are terrible and nobody ever plays will not be ranked by virtue of the fact that they are never played. Terrible files that are commonly played will still end up ranked, hence the "some measure".

[Dynam0]

Re: Stepmania Leaderboards Released!

^most elitist post I've seen in years, my god that's a cringey read

OP: "Hey guys we're releasing a beta of this neat algorithm that's going to have a ton of flaws so give us your feedback!

Community: *gives feedback*

Mina: "You guys are mean I can fix it on my own"


Show some backbone, it's going to be expected when you release a beta version of anything that people pick it apart.

[lurker]

Re: Stepmania Leaderboards Released!

even though i'm pretty sure it's unrankable i think this data point's pretty interesting:
kepo's op from midare 1 is given a higher rating than any song currently on this site (10.02)
the one dense part at the beginning is given a pattern difficulty of 33, but the ending jack is given 45

[Wafles]

Re: Stepmania Leaderboards Released!

Thanks for the info Lurker, gives us more data points to verify when we implement fixes.

[Leo137]

Re: Stepmania Leaderboards Released!

Worst way of taking feedback ever lmao

Site is looking pretty great, great job waffles & mina <3

I actually made a similar system for handling smo scores which has
never been released at public yet, i thought it may interest some of the ideas i put there, check your inbox waffles

[MinaciousGrace]

Re: Stepmania Leaderboards Released!

^most elitist post I've seen in years, my god that's a cringey read

OP: "Hey guys we're releasing a beta of this neat algorithm that's going to have a ton of flaws so give us your feedback!

Community: *gives feedback*

Mina: "You guys are mean I can fix it on my own"


Show some backbone, it's going to be expected when you release a beta version of anything that people pick it apart.

You're confusing giving feedback by method of indicating erroneous ratings and bugs and giving feedback by attempting to guess at what problems may or may not be and making judgments on the system as a whole using unfounded assumptions.

The former is healthy and useful. The latter is pointless.

I simply ask that people understand the difference.

[portalbob340]

Re: Stepmania Leaderboards Released!

Should I assume that each ranked chart is on the hardest difficulty?

[Wafles]

Re: Stepmania Leaderboards Released!

Should I assume that each ranked chart is on the hardest difficulty?

Yes. We will take special consideration to note any easier ranked difficulties. For further clarification I will likely be adding a difficulty field into the site soon. (hm, I'll probably want to call it something other than "difficulty" since that could be confusing next to "difficulty score")

[One Winged Angel]

Re: Stepmania Leaderboards Released!

pretty interesting stuff, would like to see how accurate the top players believe the difficulties are once the algorithm is refined

potentially already noted and being worked on but noticed that kirlian shores 1.4 is producing a lower difficulty rating than 1.3, but 1.5 seems to scale appropriately

[Guest15937]

Re: Stepmania Leaderboards Released!

Are the difficulties supposed to correspond to FFR divisions?

[Kyzentun]

Re: Stepmania Leaderboards Released!

In short, the algorithm makes passes on the file for assumed player difficulty "X", increasing the value by 0.01 every time. Once it gets 93% that becomes the difficulty of the file.

Please don't actually only increase skill by 0.01 each time through the file. Use a binary search instead. A few minutes of work now can save you hours of waiting later when you decide to change the algorithm and have to make it rerate all the simfiles.

[Kraezymann]

Re: Stepmania Leaderboards Released!

Are the difficulties supposed to correspond to FFR divisions?

Although it seems close, the difficulty numbers represented on the leader boards are not actually meant to correspond to division numbers on FFR. They may be sort of similar, but they stop being close after about 5 or so, I think.

These numbers are also temporary, and will always be changing as the calculators algorithms change

[Wafles]

Re: Stepmania Leaderboards Released!

Please don't actually only increase skill by 0.01 each time through the file. Use a binary search instead. A few minutes of work now can save you hours of waiting later when you decide to change the algorithm and have to make it rerate all the simfiles.

Yeah it wasn't a problem when we had relatively few files, I will be optimizing. Right now its like 10-12ish minutes to recalculate all 724 files, so yeah, as that starts getting into the multiple thousands its going to start being a problem.