New Skill Analysis Metric (Trial Phase)

[Trumpet63]

Status: Dormant
See Also: All Current Recorded Zenith Scores

zenith-score.xlsx (v.2.1.2)
If you have Excel, you can use this. Go to your level ranks page and copy the whole table, including the column titles (in Firefox, it's some kind of Ctrl+Shift with clicking and dragging from just outside the table). Paste it into B1 of Sheet1, go into Sheet2, and Sheet3, and all the results will be there. (Credit to Silvuh)

Alternatively, you can post in the thread and I will return the results to you, after which I can use your scores to improve the formula.

The Zenith Score is a measure of skill potential in units of difficulty.

This is the most recent scale by average:



Pretentious name is pretentious, I know. _Zenith_ was the first person other than myself that I used this on, and I thought it sounded cool. The graph above will be called the Zenith Curve, and the number you get from it will be called the Zenith Score. Zenith also had no idea I was posting this.

So far I'm only using Excel to do the calculations because I do not have a working knowledge of any programming language that can work with chunks of data such as this. In the end it would really only create issues with implementation if this became widely used.

Any feedback would be tremendously helpful. Having an opportunity to share this with people, and to get people excited about the enormity of the statistics here would make me very happy.

And just to give you a taste, the graph above indeed belongs to Zenith, and his Zenith Score is 72.01.

Volunteer Appreciation Here:
Tim Allen
PhantomPuppy
noname219
LethalMutiny
Callipygian
================

How the Original Works: (10 steps)

1. I copy the levelranks table (of course) and paste it into Excel.

2. Each score is sorted by the Good Count, and a value is assigned to other non-goods in the following way:
Perfect = 0
Good = 1
Average = 1.8
Miss = 2.4
Boo = 0.2
(Each of the P/G/A/M/B counts are multiplied by these values, then added)

3. Every score with 20 goods or above is removed from the table.

4. Each score is then assigned a new value, based on the following formula, hereafter referred to as the Weight.
Weight = (20-[Good Count])*[Difficulty]

5. Next, the weights for each difficulty are added together separately and arranged in a new table in the following way:
Column1 = "Difficulty" || Column2 = "Sum of Weights"

6. (At this point the graph could be made) The actual Zenith Score is obtained by first isolating all of the weights associated with the difficulty AFTER the highest weight value, or the "peak". i.e. All the values to the right of the peak.

7. Take the sum of all the values in the Sum of Weights column that have been isolated, this will be called Sum1.

8. Make another column that is the product of Difficulty and Sum of Weights:
Column3 = "DW Product" = [Difficulty]*[Sum of Weights]

9. Take the sum of all the values in this new DW Product column, this is will be Sum2 (again, only the ones to the right of the peak).

10. Zenith Score = Sum2 / Sum1

News and Progress:

9/30/14 I considered changing the Sum of Weights to an Average of Weights (the Y-value). Doing this was an effort to account for the fact that the difficulties in the game are not evenly distributed (refer to chart below). However, I found that when the sample for a particular difficulty is very small, the weight will become artificially high (i.e. one song played at D70 will imply you have AAA's on all D70 songs).



10/1/14 Thanks to the help of Silvuh, I have been able to conduct many more trials in much less time. After re-testing my Zenith Score, I found it to be 39.95, which, according to the current model of the scale, puts me borderline D2/D3. Since I was one of the last to be placed in the Tenth Official, and was placed into D3, but was previously D2, it is evident to me that I am in fact borderline D2/D3. Based on this and mr_candy and Zenth's scores agreeing with their current division placement as well, I am still hesitant to say the original formula is terribly flawed.

I postulate that a change in the division scale could be a more effective solution than developing a new calculation method. Nevertheless, I and Silvuh agree that an intuitive solution to the difficulty concentration problem would be to divide each weight by the number of songs in each difficulty. Yet more testing is necessary to rule out the usefulness of either method, and indeed I still find that the Average of Weights method conveys something with the graph that is still very powerful. For today I have to leave this issue of the three methods alone because of time constraints. I'd like to add that I don't like the idea of punishing players for playing in a difficulty range that has a lot of songs, or in effect making those scores less significant, and this is what this new counter-weight method seems to do.

10/2/14 A significant sample of 30 players has been assembled into a table to examine the viability of the Zenith Score and its variations for division placement. Adjustments to the scale should come next. In addition, two changes are to be made: 1) The length of the spreadsheet needs to be enlarged to account for the song additions caused by the Tenth Official. 2) A complete list of note counts for public songs needs to be assembled and integrated into the spreadsheet. It is likely that the note counts have to be arranged in alphabetical order because that would be the only arrangement that is consistent among all players.

10/5/14 Note counts added, sheet extended, and non-fully-played songs discounted thanks to the Excel work by Silvuh. We are adding a couple new dimensions to the Zenith Score by including new versions of the scaled score in a attempt to grasp at the concept of skill, and try to accurately predict division placements. Also the original scaled score is receiving an overhaul to correct the one-score-in-a-difficulty problem. After the spreadsheet is fully updated, we will conduct all of the trials on the table again, and afterward move into profiling entire divisions' scores. We may be looking for volunteers to conduct this large number of trials.

10/12/14 After many bugfixes in the spreadsheet, thanks to Silvuh, we can finally move into the next phase of development by making a complete survey of the tournament and its participants. Hopefully this will help us determine which of the six methods currently in use are best, and help us to gain a deeper understanding of the statistics themselves.

10/13/14 The entire tournament has been surveyed, thanks to several volunteers, and the work of Silvuh and myself. In addition, the larger portion of analysis as been completed. We have found the average, standard deviation, maximum, and minimum in each division, and for each flavor of Zenith Score. The next task is finding out the ideal scale of scores that results in the lowest error count. Current error estimations are at about 19% for W4, over all the divisions. After determining the best method to make the division scales, and finalizing the error count, the next step will be to hypothesize changes to the formula that may reduce the error count.

[mr_candy]

Re: New Skill Analysis Metric (Trail Phase)

tl:dr do me do me

[Sorie]

Re: New Skill Analysis Metric (Trail Phase)

I'd like to see my Zenith Score/Curve. I'm expecting a descending curve.

That curve..

[One Winged Angel]

Re: New Skill Analysis Metric (Trail Phase)

why is there a sudden influx of these types of threads lol

sure I guess haha

[PrawnSkunk]

Re: New Skill Analysis Metric (Trail Phase)

yes, but what about the Zenith Zenith Zenith Zenith Zenith

[Trumpet63]

Re: New Skill Analysis Metric (Trail Phase)



Your Zenith Score is: 59.82
Your best difficulty is: 45
With 3 songs under 20 goods, and an average of 2.07 goods among them

EDIT: Fixed? with updated formula

[mr_candy]

Re: New Skill Analysis Metric (Trail Phase)

Your Zenith Score is: 66.14
Your best difficulty is: 60
With 13 songs under 20 goods, and an average of 8.48 goods among them

Cool....what does that mean?

[MyNameIsNothing]

Re: New Skill Analysis Metric (Trail Phase)

seems cool, i'd like to see mine.

[Trumpet63]

Re: New Skill Analysis Metric (Trail Phase)

Cool....what does that mean?

Essentially, if your skill could be described by a difficulty level, 66.14 would be it... so far... if my system is valid at all.
It only uses songs for which you have less than 20 goods, keep in mind.

[CammyGoesRawr]

Re: New Skill Analysis Metric (Trail Phase)

do it for me~

[Trumpet63]

Re: New Skill Analysis Metric (Trail Phase)



Your Zenith Score is: 76.57
Your best difficulty is 68
With 24 songs under 20 goods, and an average of 0.6 among them.

EDIT: Fixed with the formula changes

[Trumpet63]

Re: New Skill Analysis Metric (Trail Phase)

Actually... Sorie, your scores made me rethink something.
I think instead of using the sums of the weights for each difficulty, I need to use the average because the number of songs in each difficulty vary wildly, and you've AAA'd like all of the songs below 62 (wow really?).

Recalculating your Zenith Score, I get 76.57.

I'll redo yours mr_candy, and I'll change the graph for both of you to see if it works better. (I may just edit the post to prevent bumping)

ALSO: I'm tired guys, and I have to study for my Biology test tomorrow. Let me revisit this in a day or so, after I make the above changes to the two people I did already.

[mr_candy]

Re: New Skill Analysis Metric (Trail Phase)

yeah no rush mate

[blindreper1179]

Re: New Skill Analysis Metric (Trail Phase)

If you've got the time, I'd like to see my graph. No rush.

[Trumpet63]

Re: New Skill Analysis Metric (Trail Phase)

See, I love this... although the averaging does eliminate the problems with song-difficulty concentration, it makes it very easy to get misleading information from a particular difficulty where a player may have played just a few songs but got very good scores.

I think the previous version of the system will be more valid, especially if I somehow added a weight to each difficulty that counteracts the difficulty concentration issue. I'll start working on that first... tomorrow... before mr_candy freaks out about how I put him in D4.

EDIT: THANK YOU for your patience