Bug reporting - could it be better?

thbroman · 08-07-2013, 09:13 AM

Although I've played OOTP for longer than I can remember, I believe my total contribution to bug reporting over the years can be counted on one hand - probably on Three Finger Brown's hand, come to think of it!

Anyway, I posted a bug in the thread a few days ago, too close to the release of patch 4, unfortunately, and I found the format to be pretty unsatisfactory for a couple of reasons. First, it was extremely tedious to see whether the particular bug I wanted to post (i.e., that pitchers won't be used as pinch hitters by the a.i., even when designated as the #1 choice) had been posted by anyone else. That's not surprising, perhaps, because even if bug reporting were reorganized so that each new bug received its own thread in a "bug forum," a thorough search through all the threads would consume more time than most people are willing to devote to it. I actually try to scroll through the threads in a bug forum in other games that use this structure, but I too have limits to my patience and - horror of horrors! - on occasion have posted bug reports that were already posted.

More importantly, posting bugs in a bug forum would give Markus or betas a chance to respond to individual reports, which would let the poster know that the bug has been noted, that it is being investigated, or that what's reported as a bug isn't in fact a bug. As things stand now, there's really no way for us to know whether a bug came to Markus's attention or not, a problem compounded by his habit of not providing a detailed changelog when he releases new patches.

As I said, I have been around long enough to know that other bug reporting systems have been used OOTP, for example with each report getting a "ticket" number and changelogs reporting which tickets were fixed. So it's clear that Markus has experimented with different ways of handling bug reports over the years. All I am saying is that the current system is not a very transparent one and now that most of the work on OOTP14 is done, perhaps this might be a moment to discuss whether a different bug reporting system would be better.

Cinnamon J. Scudworth · 08-07-2013, 10:54 AM

I've always assumed that the ticket system is still the default, and the bug report threads are basically just there to collect info quickly after release dates to try to make the first patch as comprehensive as possible.

I would never feel self-conscious about potentially reporting something that has already been reported by someone else. After all, it can sometimes help pin down the cause.

I doubt if OOTP team members actually have the time to respond individually to bug reports.

Déjà Bru · 08-07-2013, 11:39 AM

It's a toss-up in my mind whether it's better to have one massive bug report thread or dozens/hundreds of repetitive individual threads. I do know that advanced search can be done by post, so it is possible to find references to one's problem in the one thread.

thbroman · 08-07-2013, 12:58 PM

Quote:

Originally Posted by Déjà Bru

It's a toss-up in my mind whether it's better to have one massive bug report thread or dozens/hundreds of repetitive individual threads. I do know that advanced search can be done by post, so it is possible to find references to one's problem in the one thread.

Sure, that can be done.
My point wasn't really as much about the difficulty of searching for previous reports as it was about the difficulty of knowing whether a bug report has been taken on board or not. I realize that not every single post can be replied to, but there's a considerable distance between "every single one" and "none at all".
As for the merits of one monster thread or many, one argument in favor of the many is that it would make it easier for subsequent posters to comment/add information to earlier reports.

Fyrestorm3 · 08-07-2013, 01:30 PM

Quote:

Originally Posted by Cinnamon J. Scudworth

I've always assumed that the ticket system is still the default, and the bug report threads are basically just there to collect info quickly after release dates to try to make the first patch as comprehensive as possible.

Yeah, I've got a similar outlook. Major issues require a ticket, minor ones go in the bug report thread. The stuff I usually post in that thread is database errors - often unnoticed, easily fixable - and pstrickert always corrects them right away. If I had a major problem with the game, I'd want to open a ticket rather than post in the thread.

Déjà Bru · 08-07-2013, 03:49 PM

Quote:

Originally Posted by thbroman

Sure, that can be done.
My point wasn't really as much about the difficulty of searching for previous reports as it was about the difficulty of knowing whether a bug report has been taken on board or not. I realize that not every single post can be replied to, but there's a considerable distance between "every single one" and "none at all".
As for the merits of one monster thread or many, one argument in favor of the many is that it would make it easier for subsequent posters to comment/add information to earlier reports.

Maybe a bug status board? A table showing acknowledged issues and their statuses? [I fully realize that I am volunteering someone(s) for more work.]

endgame · 08-07-2013, 04:08 PM

Ambitious subject, but let me simplify if I may. Big changes ain't gonna happen.

A multitude of factors play into the discovery and reporting of perceived bugs and their subsequent investigation for possible resolution. A few:

User reports the issue with or without complete data, operating system, version, ability or not to replicate incident, steps to do so, specific league settings, environmental context, unique modifications implemented, the individual's league structure, etc... all which may lend themselves to the immediate identification of a core contributor, but seldom are contained in the original report AND most have to be culled through further inquiry, the submission of league files, and successfully securing answers and files.

The present behind-the-curtain reporting system is two-fold: the ticket system which is overseen by Technical response volunteers AND the bug-threads, routinely reviewed for transfer into the official framework and assignment, and culled by team members who continue their involvement beyond the finality of the initial new version's (not patches) release.

The system, as a whole, works relatively well, but of course there are many factors out of direct control of anyone but the developers. Was it really dealt with, aside from the changing of its status to Fixed? How was it dealt with? And so on, but you know what? It doesn't really matter. The only information that remains relevant on our end is, did it result in doing now what we want it to do and is it working as Markus/Andreas intended it to work? If not, back to the drawing board, submit again, try again, etc...

So I'm not trying to discourage your discussion at all, really. In fact, maybe a bit of the ol' brainstorming will stumble on to something novel and have merit that can, itself, be a new feature request and become a useful innovation. But the reality is, systemically, the present framework meets most of our needs and gets a fair amount of attention from key personnel. As timely and detailed as you'd like? Probably not, but it works as good as it can, often better than expected, with enough people involved exchanging the right amount of information. And let's face it, there's a lot more behind the scenes of a business than fine-tuning the product. The brass still do a remarkable job of acting on a game-breaker.

Here, if I can help. Carry on.

thbroman · 08-07-2013, 06:49 PM

Well, lets see what can be advanced in reply to endgame's post . . .

No doubt a lot of reported problems with the game - hangups, crashes, weird screen displays, problems with on-line leagues and the rest - depend on detailed info about the system(s) running the game, complete with dxdialog reports and so forth.

But there are also gameplay bugs that, one supposes, would actually benefit from having a separate listing and feedback from Markus or a beta. I don't want to suggest that Markus should personally reply to every single listing of this sort. Plainly that would be a waste of his time. On the other hand, if the tech support forum were organized so that one could see more or less at a glance what kind of bugs had been reported and whether they have been acknowledged, players would have a source of useful information about what they should be on the lookout for in their own games.

For purposes of argument I'll go back to the bug I reported: the a.i. won't use a pitcher who has been designated as a pinch hitter under any circumstances. That's an obvious gameplay issue which doesn't require detailed knowledge of my system, error logs, or anything else. Now, it seems to me that that's the sort of report that could generate a rapid reply from someone, on the order of "in line for the next patch," or "won't be fixed in OOTP14" or something like that. Or maybe other players could post replies confirming the issue, or contradicting it & saying that my original claim is mistaken, that pitchers ARE used as pinch hitters. As things stand now, however, it's impossible to know what's what.

By the way, I do not see this as a major problem with OOTP. Not at all. By whatever magic it happens, each new release of the game is remarkably unbuggy, crash-free, and fun. All I'm trying to do is suggest a small improvement.

Déjà Bru · 08-07-2013, 09:15 PM

Quote:

Originally Posted by Déjà Bru

Maybe a bug status board? A table showing acknowledged issues and their statuses? [I fully realize that I am volunteering someone(s) for more work.]

Quote:

Originally Posted by endgame

Ambitious subject, but let me simplify if I may.

Quote:

Originally Posted by thbroman

Well, lets see what can be advanced in reply to endgame's post . . .

What about a reply to MY post?!? Is my suggestion that dumb?

(

)

thbroman · 08-07-2013, 10:05 PM

Indeed not! It was a good idea!

Déjà Bru · 08-08-2013, 02:26 AM

Quote:

Originally Posted by thbroman

Indeed not! It was a good idea!

Thanks. Just not practical, I would guess.

08-07-2013, 10:54 AM	#2
Cinnamon J. Scudworth All Star Starter Join Date: Jul 2011 Location: Washington, DC Posts: 1,119	I've always assumed that the ticket system is still the default, and the bug report threads are basically just there to collect info quickly after release dates to try to make the first patch as comprehensive as possible. I would never feel self-conscious about potentially reporting something that has already been reported by someone else. After all, it can sometimes help pin down the cause. I doubt if OOTP team members actually have the time to respond individually to bug reports. __________________ "Sometimes, this is like going to a grocery store. You’ve got a list until you get to the check-out stand. And then you start reading People magazine, and all this other [stuff] ends up in the basket." -Sandy Alderson on the MLB offseason Last edited by Cinnamon J. Scudworth; 08-07-2013 at 10:55 AM.

08-07-2013, 11:39 AM	#3
Déjà Bru Hall Of Famer Join Date: Apr 2009 Location: Long Island Posts: 11,742	It's a toss-up in my mind whether it's better to have one massive bug report thread or dozens/hundreds of repetitive individual threads. I do know that advanced search can be done by post, so it is possible to find references to one's problem in the one thread. __________________ - Bru

08-07-2013, 04:08 PM	#7
endgame Hall Of Famer Join Date: Aug 2002 Posts: 16,842	Ambitious subject, but let me simplify if I may. Big changes ain't gonna happen. A multitude of factors play into the discovery and reporting of perceived bugs and their subsequent investigation for possible resolution. A few: User reports the issue with or without complete data, operating system, version, ability or not to replicate incident, steps to do so, specific league settings, environmental context, unique modifications implemented, the individual's league structure, etc... all which may lend themselves to the immediate identification of a core contributor, but seldom are contained in the original report AND most have to be culled through further inquiry, the submission of league files, and successfully securing answers and files. The present behind-the-curtain reporting system is two-fold: the ticket system which is overseen by Technical response volunteers AND the bug-threads, routinely reviewed for transfer into the official framework and assignment, and culled by team members who continue their involvement beyond the finality of the initial new version's (not patches) release. The system, as a whole, works relatively well, but of course there are many factors out of direct control of anyone but the developers. Was it really dealt with, aside from the changing of its status to Fixed? How was it dealt with? And so on, but you know what? It doesn't really matter. The only information that remains relevant on our end is, did it result in doing now what we want it to do and is it working as Markus/Andreas intended it to work? If not, back to the drawing board, submit again, try again, etc... So I'm not trying to discourage your discussion at all, really. In fact, maybe a bit of the ol' brainstorming will stumble on to something novel and have merit that can, itself, be a new feature request and become a useful innovation. But the reality is, systemically, the present framework meets most of our needs and gets a fair amount of attention from key personnel. As timely and detailed as you'd like? Probably not, but it works as good as it can, often better than expected, with enough people involved exchanging the right amount of information. And let's face it, there's a lot more behind the scenes of a business than fine-tuning the product. The brass still do a remarkable job of acting on a game-breaker. Here, if I can help. Carry on. __________________ "Try again. Fail again. Fail better." -- Samuel Beckett _____________________________________________

08-07-2013, 09:13 AM	#1
thbroman All Star Reserve Join Date: Jun 2002 Location: Madison, WI Posts: 867	Bug reporting - could it be better? Although I've played OOTP for longer than I can remember, I believe my total contribution to bug reporting over the years can be counted on one hand - probably on Three Finger Brown's hand, come to think of it! Anyway, I posted a bug in the thread a few days ago, too close to the release of patch 4, unfortunately, and I found the format to be pretty unsatisfactory for a couple of reasons. First, it was extremely tedious to see whether the particular bug I wanted to post (i.e., that pitchers won't be used as pinch hitters by the a.i., even when designated as the #1 choice) had been posted by anyone else. That's not surprising, perhaps, because even if bug reporting were reorganized so that each new bug received its own thread in a "bug forum," a thorough search through all the threads would consume more time than most people are willing to devote to it. I actually try to scroll through the threads in a bug forum in other games that use this structure, but I too have limits to my patience and - horror of horrors! - on occasion have posted bug reports that were already posted. More importantly, posting bugs in a bug forum would give Markus or betas a chance to respond to individual reports, which would let the poster know that the bug has been noted, that it is being investigated, or that what's reported as a bug isn't in fact a bug. As things stand now, there's really no way for us to know whether a bug came to Markus's attention or not, a problem compounded by his habit of not providing a detailed changelog when he releases new patches. As I said, I have been around long enough to know that other bug reporting systems have been used OOTP, for example with each report getting a "ticket" number and changelogs reporting which tickets were fixed. So it's clear that Markus has experimented with different ways of handling bug reports over the years. All I am saying is that the current system is not a very transparent one and now that most of the work on OOTP14 is done, perhaps this might be a moment to discuss whether a different bug reporting system would be better. Last edited by thbroman; 08-07-2013 at 09:14 AM.

08-07-2013, 06:49 PM	#8
thbroman All Star Reserve Join Date: Jun 2002 Location: Madison, WI Posts: 867	Well, lets see what can be advanced in reply to endgame's post . . . No doubt a lot of reported problems with the game - hangups, crashes, weird screen displays, problems with on-line leagues and the rest - depend on detailed info about the system(s) running the game, complete with dxdialog reports and so forth. But there are also gameplay bugs that, one supposes, would actually benefit from having a separate listing and feedback from Markus or a beta. I don't want to suggest that Markus should personally reply to every single listing of this sort. Plainly that would be a waste of his time. On the other hand, if the tech support forum were organized so that one could see more or less at a glance what kind of bugs had been reported and whether they have been acknowledged, players would have a source of useful information about what they should be on the lookout for in their own games. For purposes of argument I'll go back to the bug I reported: the a.i. won't use a pitcher who has been designated as a pinch hitter under any circumstances. That's an obvious gameplay issue which doesn't require detailed knowledge of my system, error logs, or anything else. Now, it seems to me that that's the sort of report that could generate a rapid reply from someone, on the order of "in line for the next patch," or "won't be fixed in OOTP14" or something like that. Or maybe other players could post replies confirming the issue, or contradicting it & saying that my original claim is mistaken, that pitchers ARE used as pinch hitters. As things stand now, however, it's impossible to know what's what. By the way, I do not see this as a major problem with OOTP. Not at all. By whatever magic it happens, each new release of the game is remarkably unbuggy, crash-free, and fun. All I'm trying to do is suggest a small improvement.

08-07-2013, 10:05 PM	#10
thbroman All Star Reserve Join Date: Jun 2002 Location: Madison, WI Posts: 867	Indeed not! It was a good idea!