Monday, 16 November 2026

Overthinking it: on the Limitations of Research; or, crashing the gate

 


Crunching the numbers.


I recently posted a second research article on A Fast Game, Where Do I Begin? Inspiration and intent in wargame design. I spent about four months on Where do I Begin; a little over three months conducting the survey and collating and analysing the data, and roughly five weeks writing it up, whenever I had a spare hour or two. There was some overlap with the analysis and the writing). The last two weeks I spent picking at the dataset like a buzzard with a carcass, teasing out a few more salient bits, but having blown my own deadline by nearly a month, I knew I had to let it go and put it out in the world.

Some people whose opinions I respect have commented positively about the article, which has been encouraging, but, like the first piece I researched and wrote up, The Value Proposition of Wargaming Awards - the Publishers' Perspective, I was left feeling that there was more to the story than I had managed to tease out.

This was by design: I prepared and sent out a two-question query letter because I didn’t want to discourage people from responding. Two questions felt more manageable for the time-poor; I had dropped a third question from the query letter so the effort of reply wouldn’t seem too demanding. I still think this was the right way to go. There’s no way of knowing for sure, but I suspect the returns may have been 20-30% lower than what I ultimately received.

Nearly all the feedback has been broadly positive from readers and participants alike. But one respondent raised something that had been in the back of my mind while I was writing up the analysis, only I hadn’t articulated it for myself so clearly.

“You write that you were astonished that over 75% of the designers who responded got their first effort into print. The flip side of that coin immediately occurred to me: do a large proportion of potential designers (or are you still considered a designer even if you're never published?) whose first submission is never printed give up?”

My first response to reading this wasn’t good. It felt like an attack; I can take criticism, but I was feeling a little raw from spending so much time on the project.

There are limits to what can be discovered for a short questionnaire, and by its nature, this sample group was made up of people who had gone once more into the breach with a prototype and a dream and had become published game designers. There was no way I could think of to isolate a population of failed one-time game designers who had given up after their first or first handful of rejections, except to approach random people on the street and ask them if they’d ever taken on such a Sisyphean task.

The writer went further:

‘No way to research this, of course, as by survey design and as a practical matter you have no data. [….] We'll never know, but perhaps even more remarkable is what percentage of rejected designers persevere and their later work does see the light of day?

This introduces a second factor. Everyone who responded was a published game designer. That was kind of the point of the article.

Surveys build their own limiters.

When I read through the email again, I realised what the writer was pointing out was very close to what was leaving me dissatisfied with what I’d written. I was only telling half of the story. And I hadn’t even tried to acknowledge there was more to it than I’d presented. In my defence, this was one of the last sections I wrote, the night before I posted the piece. I was tired and by then, probably not at my best, and I really wanted to get it done and up.

I wrote back, thanking the respondent for their thoughtful rebuttal. I said they were right – that I should have spent some time looking at the inverse take, that for every three designers who get their first offering into print, there’s another one who has to try again, push harder, and believe in themselves in their games that much more. In the end I thanked them for taking me to task, that it meant somebody was taking what I did seriously. And I meant it. I’m happy to say, we’re good.

Still, there must be people who create a game, maybe play it with friends or take it to a convention, maybe get some positive feedback, but never take the next step. Or they shop their prototype to a publisher or two, only to be told it doesn’t fit the company’s profile or customer base or some such and lose heart. Why should I care about people who never get past those first disappointments? Maybe I see something of myself in them.


What we're going to do about it

Anyway, here at A Fast Game, we’re about solutions. We’re also about snark and bawd, but mostly we’re about being positive and encouraging and uplifting. We talk about the games we love, and let others sing the praises – or not – of those we meet with indifference. In that spirit, we’ll be introducing a new semi-regular column, hopefully in the next month or so, that might help encourage the next round of new designers.

I’ve been kicking around an idea for another feature that I was going to pursue sometime next year, when things settle down a little more in my outside of wargaming life. Instead, I’m now rethinking the project. Rather than presenting it as another Big Project, I want to break it down into bite-sized morsels, posting a around one a month. I don’t want to say to much before I have something to present – that would be telling, and there are a few details to iron out – but I think it will make for interesting reading, and may even encourage those who got just so far but gave up to dust off their prototypes and take them to a convention or something, get them in front of some people.

Watch this space.

 

 


No comments:

Post a Comment

Overthinking it: on the Limitations of Research; or, crashing the gate

  Crunching the numbers. I recently posted a second research article on A Fast Game, Where Do I Begin? Inspiration and intent in wargame des...