[battists]

I can always empathize with both sides of this discussion. On the one hand, I totally get people wishing they could buy a bug free product. I get people wishing software companies wouldn’t release software until it was bug free. It’s very frustrating to buy a product and not have it work as you want it to.

But as someone who has worked in software and business for 30 years, a piece of software without bugs is just a pipe dream.

Computers are complex.
Hardware is complex.
Network connectivity is complex.
Operating systems are complex.
Software is complex.
Games are getting larger and larger in scope.

You put all of these things together and you start to realize the overall complexity involved. Writing a game probably involves 10, 20, 50 other pieces of software just to do the development work, all of which are made by different companies.

There’s some simple math you can do to quickly understand that “testing everything” is infeasible.

For example, let’s say a game has 5 configuration options. Ok, so now we have 5 test cases to tackle. Not so bad. But we want the game to run on windows, Mac, and Linux. So, we have to multiply that by three, so 15 test cases. But wait, there are multiple versions of each of those. So let’s say we want to test the most current and previous versions. Now we have 30 test cases.

But wait, there are different types of processors out there that could impact things. So we want to test on intel and AMD. Double that to 60 test cases. But wait, graphics cards make a difference. There’s probably 10 commonly used ones. 600 test cases. But wait, graphics cards have different versions. That’s 1200 test cases. But wait, while we were doing all of this development and testing, a new version of something came out. That’s another 800 test cases or something.

So 2000 test cases, each of which takes time to prepare, document, and execute. And I’m being conservative about graphics cards, software versions, etc.

That’s for five configuration options. A game like OOTP has THOUSANDS of different configuration permutations. If I have one option with 3 choices and another with 4 choices, that’s not 7 test cases. It’s 4 x 3 or 12 cases. Things multiply rapidly.

So mathematically speaking, to test every possible combination of software, hardware, and configuration in OOTP, it would likely take multiple man years with several dedicated resources. And without a HUGE team you would never be able to catch up with all of the new software releases etc.

It’s a business’ job to make money. Small software developers don’t make a lot of money. Markus and crew have been doing this for 20 years with <10 people, supporting multiple products. And trust me, they aren’t sitting over there in mansions. I’ve visited them.

If OOTP could only release one product every five years because they wanted to make it more bug free, well, then you just wouldn’t have an OOTP or any small developers.

All you can do as a small developer is to take a risk-based approach to testing an do your best with your limited resources and help from volunteers. The same goes for huge development shops as well. Even those giant developers have to use a risk-based approach.

When a critical bug comes up post-release it’s even harder, because now you have urgency to get a fix out there. You only have time for a fraction of the testing you’d like to do. You’re damned if you’re slow to patch, and damned if your patch doesn’t work for everyone.

This is a long rant, but hopefully it helps some people understand why software is released with bugs.




Sent from my iPhone using Tapatalk