You are doing Portfolio Allocation wrong!
13 Rules for Effective Portfolio Allocation using Optimization: Addressing Real-World Challenges
Abstract:
Portfolio optimization as instrument to determine the allocation of assets in a portfolio is often criticized for being impractical in real-world investing. In this article, we present 13 useful rules to address these challenges and make portfolio optimization more effective and user-friendly. By debunking common myths, we offer strategies that can be applied by both institutional and individual investors. The rules are designed to enhance decision-making, mitigate risks, and incorporate investor preferences, ensuring that portfolio optimization is a valuable tool for achieving long-term investment goals. Future articles will explore each rule in more detail, with real-world examples and Python code to guide implementation.
Introduction
As portfolio managers and quantitative analysts, we have often encountered reservations about the benefits of portfolio optimization to make decisions regarding the optimal allocation of assets. A common concern is that, when applied incorrectly, portfolio optimization can be more detrimental than beneficial. When used incorrectly, heuristic approaches often prove to be the superior alternative.
However, ignoring portfolio optimization altogether due to its complexity is not the solution. A lack of understanding or a poorly applied strategy can lead to poor outcomes, but with the right knowledge and methodology, portfolio optimization can be a powerful tool. The key is to approach it strategically.
In this article, we will address the common critiques of portfolio optimization and present 13 practical rules to make it more effective in real-world investing. These rules are designed to enhance decision-making, mitigate risks, and align with investor preferences. We will not limit our discussion to traditional Mean-Variance Optimization (MVO); instead, we’ll formulate rules that are valid regardless of the specific optimization approach used.
While there are many different optimization methods available, each with its own strengths and weaknesses, the true power lies in knowing how to combine these approaches to meet specific investor needs. By following these rules, institutional and individual investors alike can improve their portfolio optimization processes and move beyond outdated methods.
In future articles, we will explore each of these 13 rules in more detail, with real-world examples and Python code to guide implementation.
Square One: A plea for a clear investment process
Before diving into portfolio optimization, it is essential to establish two foundational elements:
Clearly Defined Investment Objectives
A Clear and Consistent Investment Approach
While both are crucial, we’ll focus primarily on the second element: the investment approach.
An investment approach refers to the systematic process an investor follows to make informed, strategic decisions about where, when, and how to allocate capital. This disciplined approach ensures that investments are aligned with specific financial goals, risk tolerance, and market conditions. It separates rational, consistent decision-making from impulsive, emotion-driven choices based on "gut feeling."
Each investment process is unique, and portfolio optimization must be tailored to each case. However, portfolio optimization doesn’t exist in a vacuum. It must be built upon a solid, structured investment process. Key questions like which asset classes are eligible, what investment limits or risk budgets exist, and how decisions will be made need to be answered before diving into optimization.
This brings us to Rule 1:
Rule 1: Have a clear and consistent investment approach!
Your portfolio optimization strategy must be grounded in a well-defined investment approach. Without a clear framework, portfolio optimization is like building a house on sand—it will collapse regardless of the quality of the materials or the expertise of the architect.
“Portfolio optimization is too sensitive to be used in practice”
One of the most common critiques of portfolio optimization is its perceived sensitivity, and there’s a fair amount of truth to this concern. It’s well-documented that portfolio optimization models, especially traditional ones like Mean-Variance Optimization (MVO), can be highly sensitive to small changes in input parameters, such as expected asset returns. Even slight adjustments can lead to dramatic shifts in portfolio weights—sometimes causing allocations to flip from 0% to 100%, and vice versa.
But why does this sensitivity exist?
The main culprit lies (mostly !) in the covariance matrix, which plays a central role in most optimization approaches. Two key factors contribute to the problem:
High Correlations Between Assets: Assets in a portfolio are often highly correlated, which increases the complexity of accurately estimating the relationships between them.
Noisy Data: Covariances are typically estimated using historical data, which are often noisy and unreliable, especially when the sample size is small. This results in high estimation uncertainty.
Many optimization methods rely on inverting the covariance matrix to calculate portfolio weights. When correlations are high and data is noisy, the covariance matrix becomes “ill-conditioned,” meaning it’s prone to large errors. This results in portfolio weights that are overly sensitive to small changes in expected returns, often leading to unrealistic and highly volatile allocations.
In practice, this sensitivity is particularly problematic when working with highly correlated assets or when the historical data used to estimate the covariance matrix is short or unreliable. While this is a common issue, it’s not a problem with portfolio optimization itself, but rather with the data and input parameters. Fortunately, there are approaches that are more resilient to this issue.
So, how can we address this?
Rule 2: Clean Up your Covariance Matrix!
Never use the raw sample covariance matrix directly in optimization. Sample covariances, estimated from historical time series, are often too noisy and unreliable to be used as-is. Instead, apply techniques like shrinkage or denoising to improve the accuracy of these estimates. These methods help to stabilize the covariance matrix and significantly reduce the sensitivity of portfolio weights to small changes in expected returns.
Rule 3: Structure your Assets!
Avoid using optimization on portfolios with highly correlated assets. When assets are highly correlated, traditional optimization approaches tend to fail. A better solution is to use asset clustering—grouping assets into clusters based on their correlations. The key idea is that assets within a cluster tend to be highly correlated (intra-cluster), while correlations between clusters (inter-cluster) are much lower. By optimizing at the inter-cluster level, you can reduce the impact of high correlations, leading to more stable and realistic portfolio weights.
“Portfolio optimization generates portfolios that underperform against equal-weighted portfolios”
One common critique of portfolio optimization, backed by several studies, is that optimized portfolios often underperform simple equal-weighted portfolios (1/N portfolios). The argument suggests that simply allocating an equal share to each asset might be more effective than using complex optimization techniques. But is there any real merit to this claim?
A key study often cited in support of this view is by DeMiguel, Garlappi, and Raman Uppal (2009). Their research shows that 1/N portfolios outperform optimized portfolios out-of-sample due to large estimation errors of the input parameters. However, the authors make a critical mistake—extrapolating recent returns.
In their study, the authors estimate expected returns (as input for their optimization) based on rolling estimates, which use the latest observed returns to predict future returns. This method assumes that expected returns can be estimated reasonably well from recent return data—an assumption that, both theoretically and empirically, doesn’t hold up.
If you're familiar with financial markets, this assumption might seem overly simplistic. Do you believe that markets can "heat up" due to positive investor sentiment, reducing future return prospects? Do you think that a stock with a 1000% price increase and an extremely high price-to-earnings (P/E) ratio is likely to have more moderate returns in the future? And do you think that market conditions, such as recessions or economic booms, can create different return profiles? If you answered yes to any of these questions, you're in good company. Numerous studies show that expected returns change over time, influenced by market conditions, sentiment, and macroeconomic factors. Unfortunately, the study by DeMiguel, Garlappi, and Raman Uppal overlooks these dynamics entirely.
This serves as a cautionary tale: the accuracy of your expected returns and covariance estimates is critical. As the saying goes, “Garbage in, garbage out.” If you use poor inputs, you’ll get poor results. Fortunately, we can learn from this mistake by remembering the following essential rule:
Rule 4: Do NOT extrapolate recent Returns!
Never use recent historical returns as the basis for expected returns. Don’t assume that last year’s performance will necessarily repeat. Whether you are optimizing a portfolio of broad asset classes (e.g., equity and bond indices) or individual assets (like stocks), you should rely on estimation methods that are grounded in sound theoretical frameworks. Approaches such as factor models or fundamental valuation models offer more reliable ways to forecast expected returns. Relying on recent historical data—especially from short time periods—creates an illusion of predictability and risks underperformance because markets operate in cycles. Even if you use a solid theoretical model that captures time-varying expected returns, you should be aware that predicting returns is still very difficult due to parameter instability and model uncertainty.
“Portfolio optimization leads to undesirable portfolios”
While expected returns and covariances are vital to portfolio optimization, they only capture part of the picture. Investors typically have preferences that extend beyond mere return and risk. These preferences may include ethical considerations (e.g., Environmental, Social, and Governance (ESG) criteria), liquidity requirements, or constraints on deviations from a benchmark.
To build portfolios that reflect these broader concerns, it’s essential to incorporate additional constraints into the optimization process.
Rule 5: Reflect all your Constraints!
Undesirable portfolio compositions can be avoided by imposing constraints. Does the idea of an all-stock or all-bond portfolio make you uncomfortable? Would you like to avoid excessive trading and limit transaction costs? Or, do you have sustainability preferences for your investment? Then, set constraints in the optimization to take these aspects into account. A useful exercise is to review each major asset class and identify potential extreme allocations that would violate your preferences or objectives. By doing so, you can integrate additional information—such as ESG data—into the optimization process, ensuring that your portfolio aligns with both financial and non-financial goals. Portfolio optimization, therefore, can accommodate more than just return time series.
Rule 6: Anchor Yourself!
Establish a benchmark portfolio and set a tolerable level of deviation. Every portfolio needs a benchmark, whether for performance evaluation or to guide active bets. This benchmark should reflect your long-term strategic asset allocation. As your estimates of expected returns and covariances evolve over time, so will your portfolio allocation. By using a benchmark and setting a maximum acceptable tracking error, you can ensure that your portfolio stays aligned with your long-term investment objectives and doesn’t stray too far from your desired allocation.
Rule 7: Measure your Diversification!
Use additional metrics to assess the diversification of your portfolio. Diversification is essential to any investment strategy, but don’t assume that portfolio optimization automatically ensures a well-diversified portfolio. This is another reason why optimized portfolios often underperform equal-weighted portfolios out of sample. To truly understand your portfolio’s diversification, consider metrics beyond traditional portfolio weights. Use Entropy measures, the (inverse) Herfindahl-Hirschman Index, or the Gini coefficient to quantify diversification. On a more advanced level, assess diversification based on underlying risk factors—such as economic growth, interest rates, or geopolitical events—that drive asset returns. This ensures that your portfolio is well-diversified not just in terms of asset allocation but also across the broader economic landscape.
“Portfolio optimization contains unrealistic assumptions and is not useful.”
Models and estimators can only incorporate a limited amount of real-world information. Traditional approaches, such as Mean-Variance Optimization (MVO), focus solely on the first two moments of a distribution—mean and variance—and/or assuming that financial markets follow a multivariate normal distribution. While these models have their place, they often make unrealistic assumptions about how the financial world works. Over time, more sophisticated models have emerged to address these limitations.
MVO, as the most widely recognized portfolio optimization approach, contains several assumptions that are out of step with the realities of the market. These include:
Financial markets are normally distributed — In reality, financial returns exhibit fat tails and they are far from a normal distribution.
Investors have quadratic utility — Investors do not make decisions based solely on mean-variance preferences. Their decisions are often driven by more complex psychological factors and they evaluate their outcomes relative to one or more reference points.
Rational investors with homogeneous expectations — Emotions, beliefs and psychological biases affect the decision process, in particular during crisis. Furthermore, the heterogeneity of expectations varies with the degree of information uncertainty.
Static, one-period investment horizon — Many investors do not have a pre-specified fixed investment horizon and they also consider the risk within the investment period.
No transaction costs or taxes — In a frictionless world, continuous trading and high portfolio turnover are not a problem, but in reality they are.
Investors are only concerned with variance — In practice, people suffer more from losses than they gain from equivalent gains. This is a key insight from behavioral finance, which shows that losses are perceived as more painful than the satisfaction from gains of the same size.
Behavioral finance and empirical evidence have clearly shown that these assumptions do not align with how real-world investors think and behave. However, does this mean that portfolio optimization, in general, is useless? We would argue that it does not.
Today, we have access to more advanced methods that better reflect investor behavior and market dynamics. One possibility is to consider more adequate utility functions using Kahneman and Tversky's "Cumulative Prospect Theory". Another is the use of models such as Mean-CVaR (Conditional Value-at-Risk) which offers a more realistic alternative by focusing on tail risks, which are more relevant to investors concerned with large losses rather than daily volatility. These models allow for more nuanced risk management, but be aware that increased model flexibility often comes with increased complexity and higher estimation risk.
While MVO can serve as a useful tool for developing initial insights, it should not be relied upon as the primary method for portfolio optimization. For more accurate and realistic results, we strongly recommend moving beyond MVO and adopting models like Mean-CVaR, which better reflect how investors truly perceive and react to risk. Investors are generally less concerned with everyday fluctuations (variance) and more focused on avoiding large, catastrophic losses.
Rule 8: Drop Variance as a Measure of Risk!
Use higher-dimensional risk measures, such as CVaR.
Portfolio optimization should be based on realistic assumptions. The Mean-Variance Optimization model does not meet this standard. We advocate for the use of Mean-CVaR, which more closely aligns with how investors actually perceive and react to risk, particularly focusing on tail risks and extreme events that are more meaningful than standard deviation in the context of real-world investing.1
Rule 9: Embrace the non-linear, non-gaussian Reality!
The world of financial data is inherently non-linear and certainly not normally distributed. While correlations assume linear relationships, the assumption of normally distributed returns underestimates the probability of extreme events. Whenever possible and reasonable, abandon linearity and normality assumptions in favor of more flexible approaches that account for (excess) kurtosis, skewness, and non-linearity.
“Portfolio optimization is too restrictive for my “active”and “flexible” investment approach”
Based on our experience, portfolio optimization is highly flexible, and almost any investment strategy can be integrated with it. While you may need to develop tailored solutions for specific situations—such as hierarchical clustering of portfolio elements (see Rule 3)—the flexibility to accommodate diverse approaches is certainly available.
Criticism of portfolio optimization as being “too restrictive” is common, but it’s often misplaced and can be problematic. In many cases, this criticism stems from a misunderstanding: the difference between “flexibility” and the absence of a structured investment process. A well-defined investment process acts as a blueprint in the financial world. It ensures that investment decisions are made based on a system of checks and balances, helping to prevent the impulsive, emotion-driven mistakes that often occur during market extremes—whether bull or bear markets. Without such a process, the likelihood of long-term underperformance increases significantly.
Let me be clear: When you hear this criticism as a defense against using portfolio optimization, you should consider two possibilities:
The portfolio manager lacks a proper investment process.
The portfolio manager is simply not equipped to implement portfolio optimization.
If a portfolio manager uses this argument to avoid portfolio optimization, you should proceed with caution. It could indicate a fundamental flaw in their investment approach or a lack of proficiency in portfolio management techniques.
Let’s reframe the question: Suppose you have short-term investment ideas or active views. Can you integrate these ideas into your long-term portfolio? Absolutely. In fact, there are well-established methodologies to blend short-term views with long-term strategies. Approaches like Black-Litterman allow you to incorporate your market views into an optimized portfolio, and they’re sufficient in most cases. For more complex cases, entropy-based methods can help you refine the integration of short-term predictions (also for higher moments of your return distribution) into your long-term strategy. Therefore, there’s no valid excuse for not utilizing portfolio optimization.
Rule 10: Implement your Views!
Incorporate short-term views, but don't abandon your long-term portfolio.
Many investors find it too monotonous to only focus on their long-term strategic portfolio. If you have short-term forecasts or active ideas, integrate them with your long-term strategy. Active deviations from your core portfolio are acceptable, as long as they remain within a reasonable range. Fortunately, numerous methodologies are available to seamlessly blend short-term views with a solid, strategic long-term portfolio.
“My optimised portfolio crashed although it had a low risk profile”
This scenario is something many investors experienced in 2022. Despite having portfolios with a high allocation to sovereign bonds—typically seen as safe assets—many suffered significant losses. The key issue here is that, in portfolio optimization, nearly all information about risk is encapsulated in the covariance matrix. This can be both an advantage and a pitfall.
It’s a blessing when the covariance matrix accurately reflects the true market dynamics. However, it becomes a curse when you optimize a portfolio based on covariance data that may have been valid for the last decade, only to face a sudden shift in underlying market conditions. When volatility and correlations change dramatically, the diversification benefits that once worked may disappear, leaving your portfolio highly vulnerable.
So, what went wrong in 2022?
Between 2000 and 2021, the correlation between stocks and bonds was generally negative, driven by procyclical inflation and expansive monetary policies during the 2010s. During this period, portfolios benefited from strong diversification between equities and bonds. However, with inflation rising sharply in 2022 (as negative supply shock), this dynamic shifted. The correlation between stocks and bonds turned positive, meaning that both asset classes fell together. As a result, the diversification benefits evaporated, leading to significant portfolio losses.
This situation can be avoided with careful portfolio management. The key takeaway: Your estimates of expected returns and covariance should reflect future expectations, not historical data.
Rule 11: Stress-Test Your Portfolio!
Use stress tests to assess your portfolio with alternative parameters.
Portfolio optimization doesn’t end once the weights are calculated. After determining your portfolio, you must stress-test it by considering alternative estimates of volatility and correlations to account for low-probability, high-impact events. A scenario-based approach works well here. Actively think about potential future scenarios and how they could affect your portfolio. This is especially critical when working with limited data that may not capture certain extreme events, which—while rare—are still possible.
Stress testing will push your portfolio’s diversification to the limit, revealing vulnerabilities that may not be visible under normal market conditions. This approach helps ensure that your portfolio remains resilient, even during unexpected market shifts.
“Portfolio optimization is too slow to adjust to current developments”
I strongly caution against attempting to "time" the markets. Even seasoned experts often struggle to predict short-term movements, and as a private investor, your chances of long-term success are slim if you try to time the market. If you view the stock market as a source of thrill and excitement, perhaps you should reconsider and opt for something less risky, like a time deposit. The stock market is not a place for gambling; investing should be a disciplined and often dull activity. For thrills, the amusement park is a more affordable option.
That said, portfolio optimization is not static. It adapts over time as market conditions change. While your portfolio should follow a long-term strategy, it's essential to periodically reassess the key parameters—such as expected returns and covariances. Performing a reality check on these assumptions once a year is generally sufficient. But be cautious: when you see dramatic shifts in your expected returns, ask yourself whether you're adjusting based on short-term market sentiment (which is often a poor reason) or on a thoughtful, objective analysis of future prospects (which is a sound approach).
For instance, let's consider an example of why regular checks are necessary: In the 2010s, yields were close to zero, sometimes even negative, which made bonds an unattractive asset. Today, yields have risen significantly, with bonds offering returns of 3% or more in Europe. This shift dramatically alters the attractiveness of bonds and may justify a higher allocation to them in the coming decade compared to the previous one. Therefore, your optimal portfolio in 2025 may look quite different from one designed 10 years ago.
Additionally, as market conditions evolve, your actual portfolio will naturally deviate from the original optimized version. This brings us to two crucial rules for maintaining an “optimal” portfolio.
Rule 12: Conduct regular Reality Checks!
Evaluate your portfolio and its parameters regularly.
Expected returns and covariances are not fixed; they change over time as market conditions evolve. Structural shifts in the economy can significantly alter risk and return profiles. Make it a habit to reassess your estimates—annually is generally enough—but be cautious not to make these adjustments based on emotional reactions to short-term market movements. Regular assessments ensure that your long-term strategy remains relevant and optimal for your purpose.
Rule 13: Rebalance your Portfolio!
Establish clear rules for when and how to rebalance your portfolio.
Remember, you designed your portfolio with a specific strategic allocation for a reason. Over time, market fluctuations will cause your portfolio to deviate from its optimal weights. To maintain the intended weighting, you should regularly adjust your portfolio in order to counteract over- or underweightings caused by market fluctuations. While the exact timing depends on your portfolio’s specifics, in most cases, quarterly or semi-annual rebalancing is sufficient. This process can also generate additional returns, as it often results in "buying low" and "selling high," which can be beneficial in certain market conditions.
By implementing these regular checks and rebalancing routines, you ensure that your portfolio remains aligned with your long-term goals while adapting to changing market realities.
Summary
In this article, we've outlined 13 practical rules for optimizing portfolios in real-world settings. By addressing common misconceptions and challenges—such as the sensitivity of optimization models and the dangers of extrapolating recent returns—we've provided feasible solutions that both institutional and individual investors can follow to improve their portfolio management processes. Key recommendations include using robust covariance estimations, incorporating investor preferences through constraints, and adopting advanced risk measures like the Conditional Value-at-Risk (CVaR) over traditional variance.
These rules are grounded in both theoretical finance and empirical research, offering a solid framework for enhancing portfolio optimization beyond the limitations of traditional approaches like Mean-Variance Optimization. In future articles, we'll delve deeper into each rule, providing real-world Python examples to guide you through implementation.
Portfolio optimization, when done correctly, can be a powerful tool for building resilient and strategically aligned portfolios. With these 13 rules, we hope to help you navigate common pitfalls and make better-informed investment decisions. Stay tuned for more detailed discussions in the upcoming posts.
We highly advise you to have a look at the Anton Vorobets substack:
https://antonvorobets.substack.com/t/articles
 | A guest post by
|