Model Complexity Considerations

Understanding Model Complexity in ConversionFlow

The ConversionFlow library implements Bayesian Network models of varying complexity to capture the intricacies of customer journeys. This document explores the considerations around model complexity, including how to balance complexity with interpretability and computational efficiency.

Dimensions of Model Complexity

Structural Complexity

The structural complexity of a ConversionFlow model is determined by:

  1. Number of Nodes: Each touchpoint in the customer journey adds a node to the network. More nodes allow for more detailed journey mapping but increase computational demands.

  2. Edge Density: The proportion of possible edges that are actually present in the DAG. Sparser networks are easier to interpret and compute, while denser networks capture more complex interdependencies.

  3. Node Types: The current implementation uses primarily binary nodes (whether a touchpoint was engaged with), but future extensions might include:

    • Continuous nodes (e.g., time spent on a page)

    • Count nodes (e.g., number of visits)

    • Categorical nodes (e.g., device type)

Parametric Complexity

Beyond structure, models can vary in parametric complexity:

  1. Prior Distributions: More flexible priors like the Half-Cauchy (currently used) allow for a wider range of parameter values compared to more restrictive priors like the Half-Gaussian.

  2. Parameter Sharing: Some models might share parameters across similar touchpoints to reduce complexity.

  3. Non-linear Relationships: The logarithmic term for budget effects introduces non-linearity, but additional non-linearities could be introduced for more complex relationships.

Computational Complexity

The computational demands of model fitting and inference scale with:

  1. Number of Parameters: More parameters require more MCMC samples for convergence.

  2. Graph Connectivity: Highly connected graphs create complex parameter dependencies that slow MCMC sampling.

  3. Data Size: Larger datasets provide more information but increase computation time.

Model Selection Strategies

Cross-Validation

ConversionFlow implements cross-validation techniques to compare models of different complexity:

  1. Leave-One-Out Cross-Validation (LOO-CV): Implemented via the az.loo() function from ArviZ to estimate out-of-sample predictive accuracy.

  2. Widely Applicable Information Criterion (WAIC): An alternative to LOO-CV that approximates Bayesian cross-validation.

The model with the lowest LOO or WAIC value typically represents the best balance of fit and complexity.

Posterior Predictive Checks

Beyond formal cross-validation, ConversionFlow employs posterior predictive checks to assess model adequacy:

  1. Graphical Checks: Comparing the distribution of observed data to predictions from the model.

  2. Statistical Discrepancy Measures: Quantifying differences between observed and predicted data.

  3. Specific Test Statistics: Targeting particular aspects of the data that should be captured by the model.

Balancing Complexity and Utility

When to Increase Complexity

Consider a more complex model when:

  1. Simple models show poor predictive performance in cross-validation.

  2. Domain knowledge suggests important relationships are missing.

  3. Posterior predictive checks reveal systematic discrepancies.

  4. The business context requires more detailed insights.

When to Reduce Complexity

Consider a simpler model when:

  1. Parameter estimates show high uncertainty (wide posterior distributions).

  2. MCMC sampling exhibits convergence problems.

  3. The computational cost becomes prohibitive.

  4. The model becomes difficult to interpret and explain to stakeholders.

Practical Recommendations

Based on experience with the ConversionFlow library, we recommend:

  1. Start Simple: Begin with a sparse DAG structure focusing on the most important touchpoints.

  2. Iterative Refinement: Gradually add complexity while monitoring cross-validation metrics.

  3. Domain Knowledge Integration: Use business understanding to guide structural choices rather than relying solely on data-driven approaches.

  4. Sensitivity Analysis: Assess how sensitive optimization recommendations are to changes in model complexity.

Case Study: Evolution of ConversionFlow Models

The ConversionFlow library has evolved through several model iterations:

  1. Initial Model (v1): A simple linear funnel with few touchpoints and standard normal priors.

  2. Intermediate Model (v2): An expanded DAG with more touchpoints and Half-Gaussian priors.

  3. Current Model (v3): A comprehensive DAG with specialized touchpoint types, Half-Cauchy priors, and more sophisticated handling of budget effects.

Each iteration has brought improvements in predictive performance and business utility, but also increased computational demands.

Future Directions in Model Complexity

Future developments may explore:

  1. Hierarchical Models: Allowing for customer segmentation with segment-specific parameters.

  2. Time-Varying Parameters: Capturing how touchpoint effectiveness changes over time.

  3. Structural Learning: Using causal discovery algorithms to learn optimal DAG structures from data.

  4. Neural Network Integration: Combining Bayesian Networks with neural networks for more flexible functional forms.

These advances will need to be carefully balanced against the core values of interpretability and computational tractability that make ConversionFlow valuable for practical marketing decisions.