Queue/Batch Discussion Thread

[Coolboyrulez0]

Re: Queue/Batch Discussion Thread

FFR needs more jumpstream and chill out on the bursty dnb songs for a bit. lol.

Agreed. We also need more songs in VD-VC range imo. (and in the extremely easy range)

[Izzy]

Re: Queue/Batch Discussion Thread

Agreed. We also need more songs in VD-VC range imo. (and in the extremely easy range)

I would have to agree as well. I find 7-10's a lot of fun. Those are the kind of songs I like playing in multiplayer.

I think one of the problems is that most of the people stepping songs are stepmania veterens who are really good at the game. A song isn't stepped right unless you have a note to every noise in the song.

[TC_Halogen]

Re: Queue/Batch Discussion Thread

The only reason I can think of is that perhaps decisions may have been finalized based off of Halogen's earlier post, which in this case I do notice a discrepency (noted as +.,+.,+. = 12pts). I'd like to have jx confirm this.

Human error sucks, I know.

Did I do something wrong with how the ratings were accumulated? From what I see here, everything looks fine. =/

[bmah]

Re: Queue/Batch Discussion Thread

Read 0's post. Patashu rated the song as a [+]. So it should be 13 pts.
@Izzy: I disagree with that, and Patashu has already made some deliberately understepped easy files.

[jimerax]

Re: Queue/Batch Discussion Thread

I think majority of files in the game are 8-10s already.

[dore]

Re: Queue/Batch Discussion Thread

@dore: the inconsistency due to the level of judging experience (or lack of) is something I'm trying to tackle...it's one reason why I made a progress report, as well as providing resubmitted files an extra sort of distinction amongst the others. I have a number of files that weren't accepted that I suspect was due to this issue. If I actually can't find anything to fix, I'm going to send them completely unchanged, because it's not an issue on my end if it's due to judging incompetency. You can't completely eliminate the problem with this many judges, but I'm trying to reduce it.

I see where you're coming from. I think fundamentally where I disagree is the numerical approach, but that's probably more because I'm used to the Tass system. I don't know if having a few miniature Tasselfoots would be any better than a numerical approach, but I guess I feel like the judging-the-judges shouldn't be completely objective because that overlooks the fact that judging is inherently subjective and, especially with the discrepancies in experience between these judges, I think that it's difficult to really guess where each of the judges are in their judging without more information or more direct comparisons.

[bmah]

Re: Queue/Batch Discussion Thread

Yep. The objectivity introduced by a numerical system just makes decisions a bit easier, but clearly has its own shortcomings. I don't expect to find a perfect system, so it's going to have to be an imperfect one with some manual patch-up fixes to tie up loose ends.

[dore]

Re: Queue/Batch Discussion Thread

Yeah. If there's one thing that should be changed, I think there should be at least a little gray area somehow.

[jimerax]

Re: Queue/Batch Discussion Thread

Even though there're things that could be improved but I don't find this system any worse than old system (too much work for judges, final decision was biased).

Putting secondary judges for borderline files would make the judgement quality better, it will be more complicated and slower though.

[TC_Halogen]

Re: Queue/Batch Discussion Thread

11ELEVEN: [+.], [+.], [+.] (12)

My fault: should be 13. Putting all of that stuff together was a bit of a strain on the eyes, sorry about that mistake. :/

[qqwref]

Re: Queue/Batch Discussion Thread

Some minor suggestions about the grading scheme:

- If 12 is going to be a rough cutoff for acceptance, that means that a single [?] (or worse) means a file gets rejected. So there isn't really any point in having [-] or [--] except to be personally mean to someone.

- Similarly, if [++] and [+] are worth the same amount, there isn't really any point to having [++] except to be personally nice to someone. The judging criteria say that if someone gives a file [++] it will almost certainly get good ratings from other judges, but this doesn't matter because we can *see* ratings from other judges.

- OK, now we know that some judges don't seem to give out the mid-range grades as much as they should, plus the ratings look similar enough that it is possible to get confused on what exactly people mean. I think this might be due to confusion since they look similar. So I suggest changing the 6 grades (5/5 to 0/5) from this:
[+], [+.], [+?], [?], [-], [--]
to the more clear:
[++], [+], [?], [-], [-], [-].

[jimerax]

Re: Queue/Batch Discussion Thread

Now I think judgement rating should be 5/5 - 1/5 instead of +?- things, since it looks like half of judges aren't accustomed to current ones (and got more inconsistent because of that). Of course it would give a bad effect if some judges only use + and - because they don't know how to use intermediate ratings.. ok for judges who are used to old one though.

++ and -- are the same with + and - score-wise but we will take judge's intention at least.

[dore]

Re: Queue/Batch Discussion Thread

Another random idea:

In the old system, the difference between + and +. was the difference between queue and conditional queue, usually. I took the distinction between the two was that + means that judge wants that file queued as is, and +. means that judge would like small changes but overall it's acceptable as is.

To me, the distinction between + and +. is much smaller than that of +. and +?. To me, +? was a file that had potential but needed changes to be acceptable.

It seems to me, then, that + and +. could be the same point value for acceptance.

+ = 4 points
+. = 4 points
+? = 3 points
? = 2 points
- = 1 point

Then you could just have a hard cutoff at 11+ = accept, <11 = reject, with maybe the exception that two +s is an auto-accept, regardless of the point value (so that a +,+,- or +,+,? still goes through). The judges could then confer and decide the difference between queue and conditional queue depending on whether the judges' notes agree or not.

just throwin around ideas to debate

[Korygon]

Re: Queue/Batch Discussion Thread

I was thinking potentially something like

+ and ++=3
+.=2.5
+?=2
?=1
-=0

With 8 or more being accept, and <8=rejected

[TC_Halogen]

Re: Queue/Batch Discussion Thread

why use a decimal to complicate things?

Simply put: do a 5 point scale, three judges, 12 point acceptance minimum with discretion used on files that fall below 12 points due to a single judge's difference in opinion. Forget all of the symbol nonsense.