Queue/Batch Discussion Thread

[TC_Halogen]

Re: Queue/Batch Discussion Thread

I didn't say anything because it was her file, I said it because I disagreed with some aspects, especially since I actually helped with certain parts of the file. I also fixed things that I did agree with, so there's no need to make hasty assumptions when all I'm trying to do is help.

I too also said that I disagreed, just like how judge opinions don't line up with anyone else's, which is why I asked each individual judge to get input from everyone. If I was extremely desperate to get Kayla's file in game, I would have [++]'d it. But guess what, I'm actually honest and looking for her to get improvements.

It's also not a waste of time considering it's the batch discussion thread, and I took apart everything for the reason of discussion, not for argument.

I also think it's about time I make a thread that I've been wanting to make for a couple of weeks now.

just suggesting: files with two +s could be borderline and reviewed? This comes from a particular example from my group (Feux D'Artifice) which I think has no room for improvement and only didn't make it because of one outlier judge (+ + +. -). I only see one other file that this would apply to, so whatever.

Files like these should be passed around to other judges for a larger sample of rating data.

[bmah]

Re: Queue/Batch Discussion Thread

Wow, The Final Hour is the most hotly contested file ever. lol

[i love you]

Re: Queue/Batch Discussion Thread

Wow, The Final Hour is the most hotly contested file ever. lol

^^

[jimerax]

Re: Queue/Batch Discussion Thread

just suggesting: files with two +s could be borderline and reviewed? This comes from a particular example from my group (Feux D'Artifice) which I think has no room for improvement and only didn't make it because of one outlier judge (+ + +. -). I only see one other file that this would apply to, so whatever.

files that 2 +'d and 3 +.'d.

Sing for Absolution (Patashu) - 16 (4/2/5/5)
Feux d'artifice (Xandertrax) - 15 (4/1/5/5)

You (judges) can discuss about the reasonability of the ratings (lower ones).

[bmah]

Re: Queue/Batch Discussion Thread

Couldn't I say the same about other files as well? I mean, I'm looking at several files here that were borderline from all groups, and any of them could've EASILY got in if one or two reviews from each group wasn't so picky about minor things (i.e. +, +., and +? really do make a difference here, nevermind other ratings). If you actually find an error on the judge's part (which shouldn't happen too often) which made the difference between an accept or a reject, please report it. It's your file after all.

[psychoangel691]

Re: Queue/Batch Discussion Thread

Couldn't I say the same about other files as well? I mean, I'm looking at several files here that were borderline from all groups, and any of them could've EASILY got in if one or two reviews from each group wasn't so picky about minor things (i.e. + and +. really do make a difference here).

That was the whole point of his post. That's what I'm trying to point out he said at the end of it

The point that I'm trying to make is, people are judging way too harshly on extremely subjective errors and not enough on objective errors like missed sounds, progression errors, etc.

The final hour was an easy example for him to use because he had gone through it with me before I sent it in to help look for errors. He felt the same way about some judgements in the last batch.

[dore]

Re: Queue/Batch Discussion Thread

I kinda feel like the best solution might be to throw out the top and bottom scores, but we'd have to have 5 judges per team at that point, and there's no way to do that without either more qualified people (which we don't really have), or increasing everyone's workload (which shouldn't be necessary).

[i love you]

Re: Queue/Batch Discussion Thread

Also, what was the point of deleting your post and then reposting the same thing psychoangel?

Just curious, it was rather random lol

[bmah]

Re: Queue/Batch Discussion Thread

That was the whole point of his post. That's what I'm trying to point out he said at the end of it

The final hour was an easy example for him to use because he had gone through it with me before I sent it in to help look for errors. He felt the same way about some judgements in the last batch.

Then what we're looking at here are conditionally queued files. Right now, it appears that we have no physical definition for determining what a conditionally queued file is - the exception at the moment simply appears to be only offsync files. In the past, conditional queue was based whatever Tass felt was conditional. Here, I think we need to more concretely define "conditional".

Once defined better, I hope such files will be included in a conditional list. Take note of this jx.

[psychoangel691]

Re: Queue/Batch Discussion Thread

Wait wtf, what was the point of deleting your post and then reposting the same thing...

I didn't mean to the comp fed up and screwed something up.

I kinda feel like the best solution might be to throw out the top and bottom scores, but we'd have to have 5 judges per team at that point, and there's no way to do that without either more qualified people (which we don't really have), or increasing everyone's workload (which shouldn't be necessary).

The problem is there's just too vast a difference. There's no real guidelines on what should be errors and knock points off and what's not. So you'll have one judge looking at something in a subjective way knocking a bunch of points off for it and another only looking for "actual" errors.

I think maybe if there were some sort of guidelines it might help a little bit. I mean like you shouldn't knock off for song choice, layering and pattern choices (if they are correct)

It just seems there's no general standard as to what's actually an error and what's not. Which causes such a vast difference in the judging making things more difficult. I mean look at poor kjwkjw one of his files got a + but also got a - that's pretty confusing if you ask me. From a newer stepper pov it's really confusing to understand when you have 4 different judges telling you different things to know what's actually wrong and what's not. I went through everything that was said and was confused as hell on some of it. And I tried some of the changes and it made parts of the file feel awful.

[qqwref]

Re: Queue/Batch Discussion Thread

I have an idea about judging with 4 judges:
- Out of the four, figure out acceptance by looking at the TOP THREE scores. This way, one judge cannot ruin a file.
- Figure out whether a file makes queue or conditional queue by looking at the LOWEST score (or lowest two). This way, files that can be considered to have significant mistakes can be fixed, and files that nobody has a problem with can be used as is.

Or maybe I'm just annoyed that Famicom Selecta isn't even getting conditional queue'd... haha

[bmah]

Re: Queue/Batch Discussion Thread

haha, and I thought we already fixed the "one judge's discrepency ruins a file" issue

guess not ?_?

[jimerax]

Re: Queue/Batch Discussion Thread

If one rating is totally different from others the rating needs adjustment, but I still don't want to disregard bottom(or top) ratings completely (depends on the reasonability of the comment).

also, 4 judges/team is a lot better than 3 judges/team in terms of judgement accuracy, so we'll go with this system if possible.

[TC_Halogen]

Re: Queue/Batch Discussion Thread

The amount of points needed to get by needs to be dropped. 4 [+.]'s should not get rejected by any means.

[bmah]

Re: Queue/Batch Discussion Thread

@qqrwef: I'm not afraid to admit it, when I complain it's often due to my own files haha, don't worry about that

@jx: I figure that if we can't closely discriminate a specific guideline, which I think is quite the possibility due to the ridiculous amount of subjectivity involved, a file can be contested, and you would have to provide the appropriate reasoning for rejection. If not, there may be negotiations to change that particular file. This can lead to a lot of inquiries, but it does deal with each unique case.

p.s. didn't I mention a complaint form before? yup


p.s.2: @jx: Similarly, a drawing line can be contested, such as what Halogen just did. You can drop or raise the drawing line by however many points as negotiated. This achieves a similar result to what I've already suggested, except it's less accurate. You wouldn't be reviewing each individual file that is debated.