Why Your A/B Tests Fail in Your Online Communities | #MetricsMonday

The reason your metrics and A/B tests tend to fail as you scale them is because your communities do not follow a consistent and scientific marketing funnel. Instead, they follow the core value of social science: It's not the metrics that changed - they're still true for the audiences you tested. It's the 'scope' or the environment you are implementing your A/B tests that changed.

This is why your metrics fail and we call this the 'law of numbers.' 

In a traditional scientific perspective the more datapoints you have the more accurate your data will get. unfortunately for most social scientific work this does not happen because opinions do not generally 'converge' on a general concensus. They 'diverge' until the data has stopped representing new things. 

So when your community's opinions have stopped becoming new, only THEN can you measure your community using traditional marketing tactics.

Community is a social-scientific entity. So you have to measure it, social scientifically. 

A scope is a sub-unit of community culture where like-minded individuals gather together into individual small group units, build strong bonds, and then interact in stratified spaces.