Cohort testing. For users signing up between T+0 and T+2 weeks (or whatever), you randomly assign them to N test groups. Each of the N groups has some combinations of features turned on or off, controlled via feature flags. You collect aggregate statistics over time of the conversions in each test group. Presumably, if you want to track outgoing clicks, you have the page intercept clicks and fire a Javascript event with an anonymous ID number which your backend will score as a click for the appropriate test group. The technical implementation of this will take you between 2 and 5 days or less.
The math gets hairy and you need a metric truckload of time/traffic to do anything useful with this, especially as you get to many types of conversions of interest or many different variables under consideration, as the nature of statistics means you'll always be able to find some correlation if you look in enough places. "Look doing Signup Form B and disabling private messaging decreases friend requests by 18% in month 3 with 99.999% probability!" [+]
+ This is not actually what the math is telling you.
Cohort testing. For users signing up between T+0 and T+2 weeks (or whatever), you randomly assign them to N test groups. Each of the N groups has some combinations of features turned on or off, controlled via feature flags. You collect aggregate statistics over time of the conversions in each test group. Presumably, if you want to track outgoing clicks, you have the page intercept clicks and fire a Javascript event with an anonymous ID number which your backend will score as a click for the appropriate test group. The technical implementation of this will take you between 2 and 5 days or less.
The math gets hairy and you need a metric truckload of time/traffic to do anything useful with this, especially as you get to many types of conversions of interest or many different variables under consideration, as the nature of statistics means you'll always be able to find some correlation if you look in enough places. "Look doing Signup Form B and disabling private messaging decreases friend requests by 18% in month 3 with 99.999% probability!" [+]
+ This is not actually what the math is telling you.