Posts Tagged data

Weighing Sources

Although I am filing this one under game design (it does apply, and it’s a theme!), I think the lesson applies to any endeavor where data is collected – you need to account for how much the data is worth by source, especially in context. For example, if I am playtesting a game product, my personal game biases will affect my feedback and my differential from the intended audience de-weights my feedback on resonance or feel, but not on mechanics (because I may understand game design better than the average intended audience member, for example).

I divide this into two camps: expert data and user data.  When you ask a web developer to use your site, you are usually looking for a professional opinion.  That web developer is probably a person who also browses the web in his or her spare time, but that doesn’t mean you necessarily want the feedback a “normal browsing person” would give.  In games, there are a lot of behaviors that can be predicted by good game designers and therefore they are great resources when seeking information about what to change.  I think of expert data as rolling up over time, giving you a better and better picture of an ideal design.

User data is the more prevalent type when doing studies, because anyone who can be measured is generally speaking a user.  I find it very important to weigh data in this category based on the “match” of the user in question to the “ideal audience member”.  We do this sort of implicit weighting a lot when we have arguments about game design at work, because Person #1 says they had a bad experience trying a new game component, and Person #2 mentions that Person #1 is “not that kind of player” and so their opinion isn’t really the “common person” opinion.  I think of user data as constantly shifting the game design around until you finally match some aspect of the game to some aspect of the audience.  Then you do everything you can to hold that part constant while you try again.

Data is so valuable, but misunderstanding data can be worse than none at all.

Tags: , , , , , ,

A Metrics World

A former co-worker (thanks Elf!) posted the following link on his FB:

http://petewarden.typepad.com/searchbrowser/2010/02/how-to-split-up-the-us.html

What an amazing undertaking of data analysis!  This “study” corresponds quite closely to two things I was thinking about this morning: specifically, what a time-history plot of my use of tags on this blog would look like (see this graph of last.fm usage for something akin to what I was thinking of), and what a time-history plot of people logging on/off of chat clients (which I am seeing minute-to-minute on the side of my screen right now in digsby) might look like over the course of an average day.

I have also had multiple discussions with game designers recently where the foremost on their mind was getting “real” user feedback from analyzing their metric data – their usage of the game system.  How useful is the world according to metrics?  I love that we can do interesting visualizations of the HUGE amount of data being generated on the Internet every second by millions of people… but how useful is the data itself with regard to understanding people’s behavior and/or making decisions about how to interact with people?

It reminds me of my musings on Asimov’s hypothetical field of psychohistory, and also some fundamental ideas about how individuals differ from populations.  It is a well-known pitfall in analysis that when you come up against the barrier of too many dimensions (meaning, tons of different ways to gain perspective on the data), that by choosing a perspective, you are forcing yourself into limited usefulness of results.  What I mean by this is, you can’t answer every question by looking at the data from one perspective – seems obvious, I know!

I worry that if the preponderance of accessibility to this raw metric data increases as it has been recently, that particular perspectives may gain undue weight and skew decisions toward something akin to a majority rule of perspective.

Tags: , , , ,