I’m sure everyone reading this has heard constant speeches regarding for the need of beta-testing, so I will spare you further lecturing on that subject. However, what should you look for when selecting beta-testers? As someone with a fair deal of experience, perhaps my thoughts will be useful.
Some people enjoy beta-testing, at least when the scenario is half-decent. Others dislike the process, under any circumstances. And while I certainly fall into the latter category, the majority of the BOE community seems to fall into the former. Given this, you’ll generally find no problems attracting beta-testers whenever you have the need. So much so, in fact, that you’ll have to be selective among those you choose. I make my choices based on three broad categories:
- Qualifications of the tester:
I break this down into two broad categories — competence and diligence. Obviously, competent beta-testers are preferred, as individuals who will not recognize errors when they see them are of no help at this stage. This becomes increasingly important as your skill with the editor increases. As your designs become more intricate and complex, a good beta-tester should be able to recognize the basic framework of the design, and be able to experiment with the design, to see if there are any flaws around the edges. Pre-selecting based on competence is, obviously, somewhat difficult, but I generally see experience in BOE (whether extensive playing or designing) or past experience with that particular beta-tester as the best measures.
Diligence is far less complicated, but no less important. If a beta-tester is to be of any help, (s)he must take the task at least somewhat seriously, and make timely reports to the designer. For each of my first three scenarios (Algiers/Kalloskagathos/Good&Evil), I had beta-testers vanish into the wind in the midst of testing. This quality may be even harder to prejudge, but, along with past experience, I find you can get a decent feel for the dependability of various community members if you are observant.
- Diversification:
When selecting multiple beta-testers, you should try, as much as possible, to generate a diverse selection, so a more thorough testing, bug-correction procedure can take place. Obviously the most important diversification is the Mac/PC split, as BOE performs (slightly) differently on different machines. But try to go beyond this as well — get designers and non-designers, as each will have a unique perspective on your scenario. I find it particularly important to get at least one grammar fiend, as I’m fairly incompetent in that respect. Different playing styles are also important, as your scenario might be too hard or too easy with only a 1 PC party, for example.
- Community Qualifications:
Given that most players consider testing decent scenarios to be a reward, I try to reward those potential testers who have contributed to the community. Far to many BOE community members tend, for whatever reason, to contribute only to the meekest extent possible, and I so I try to select those who break from this pattern, both as a reward of sorts, and as encouragement for more to follow their example.
Alright, so now you have some idea how to select testers, you’re probably wondering how many you should select? My general feeling is that the optimal number is two for very short scenarios, three for scenarios of medium length, and perhaps four to five for long scenarios.
Finally, let me end on the note that, as a designer, you have responsibilities as well. First, do not release your scenario for beta-testing until you yourself have finished the alpha testing phase. Play through your scenario! At the very least, you should play through — and win — your scenario prior to release, just to make sure that it’s possible. When your testers report fatal bugs, you should provide timely corrective updates, and as your testers request input, your responses should be timely there as well. Finally, even though your testers may view this as a privilege, do not ever forget that they are the ones doing you a favor! Treat them accordingly.
|