Dark energy and the accelerating universe

The expansion of the universe, first confirmed by Edwin Hubble in 1929, was long expected to be gradually slowing under the mutual gravitational pull of all the matter within it. In 1998, however, two independent astronomical teams using distant exploding stars as cosmic measuring sticks arrived at a startling conclusion: the expansion is not slowing down at all. It is accelerating.^{1, 2} This discovery overturned a foundational assumption of cosmology and introduced a new term into the scientific vocabulary — dark energy — to describe whatever is causing the universe to fly apart at an ever-increasing rate.

Dark energy is now understood to be the dominant component of the cosmos. According to the most precise measurements available, it accounts for approximately 68% of the total energy content of the universe, with dark matter comprising roughly 27% and all ordinary matter — every star, planet, gas cloud, and living thing — making up only about 5%.⁷ Despite its apparent ubiquity, dark energy remains one of the deepest mysteries in all of physics. Its nature, origin, and precise behavior are unknown, and theoretical attempts to explain it have produced what is sometimes called the worst prediction in the history of science.

Type Ia supernovae as standard candles

The key to measuring the expansion rate of the distant universe lies in finding objects of known intrinsic brightness. When astronomers know how bright something actually is, they can calculate how far away it is by measuring how faint it appears from Earth — just as one can estimate the distance to a streetlight by how dim it looks. Objects with reliably known luminosities are called standard candles.²⁵

Type Ia supernovae are thermonuclear explosions that occur in binary star systems when a white dwarf accumulates enough mass from a companion star to trigger a runaway carbon fusion reaction. Because the detonation always occurs near the same critical mass threshold (approximately 1.4 times the mass of the Sun, the Chandrasekhar limit), the peak luminosity of these explosions is nearly uniform.²⁰ After applying a correction for the observed relationship between peak brightness and the rate at which the supernova fades — brighter supernovae fade more slowly — astronomers can standardize Type Ia supernovae to within about 10% accuracy in distance, making them the most precise standard candles available for cosmological measurements across billions of light-years.^{25, 20}

By the mid-1990s, two teams had independently organized systematic searches for distant Type Ia supernovae. The Supernova Cosmology Project, led by Saul Perlmutter at Lawrence Berkeley National Laboratory, and the High-Z Supernova Search Team, led by Brian Schmidt at the Australian National University and including Adam Riess at Harvard, each set out to use distant supernovae to measure how fast the expansion of the universe was decelerating.^{1, 2} Both teams expected to find a slowing expansion; the only question was by how much.

The 1998 discovery

The logic of the measurement is straightforward in principle. If the universe is decelerating, distant supernovae should appear slightly brighter than a simple linear extrapolation would predict, because the expansion was faster in the past and they are somewhat closer than naive models would suggest. If, on the other hand, the universe is accelerating, distant supernovae should appear slightly dimmer, having been carried farther away by the extra push of acceleration during the time their light was traveling to us.¹

The High-Z team's result, published in September 1998 in The Astronomical Journal, was unambiguous. Their sample of 16 high-redshift Type Ia supernovae were systematically fainter than expected — too faint to be explained by dust, stellar evolution effects, or any systematic error the team could identify. The universe was not decelerating. It was accelerating.¹ Riess and his collaborators found that the best-fit cosmological model required a positive cosmological constant or some equivalent dark energy term with an energy density of roughly 0.7 times the critical density of the universe.¹

Perlmutter's Supernova Cosmology Project, publishing its full analysis in 1999 using 42 high-redshift supernovae, confirmed the finding independently. Their measurements indicated ΩΛ ≈ 0.72 and Ω_m ≈ 0.28, a universe in which a cosmological constant strongly dominates over matter.² The convergence of two competing teams on the same unexpected result, using overlapping but independently assembled datasets and different analysis pipelines, gave the scientific community high confidence that the discovery was real.²⁰

In 2004, further support came when the Hubble Space Telescope detected supernovae at redshifts above z = 1, corresponding to distances of more than 8 billion light-years. At these distances, the universe was young enough that dark energy had not yet come to dominate over matter, and the expansion should have been decelerating. Those supernovae indeed appeared brighter than the nearby ones — exactly as the cosmological constant model predicted, tracing the transition from a matter-dominated decelerating phase to the current dark-energy-dominated accelerating phase.⁸

In October 2011, the Nobel Prize in Physics was awarded jointly to Saul Perlmutter, Brian Schmidt, and Adam Riess "for the discovery of the accelerating expansion of the Universe through observations of distant supernovae."³

Einstein's cosmological constant

The simplest and mathematically most natural explanation for dark energy is a term that Albert Einstein himself introduced into his field equations of general relativity in 1917. Einstein added a constant Λ (lambda) to his equations to counterbalance gravity and produce a static, unchanging universe, as he believed the universe to be at the time.¹⁹ When Hubble's observations in 1929 demonstrated that the universe was expanding, Einstein abandoned Λ, reportedly calling it his "greatest blunder." The 1998 supernova discoveries resurrected it.

In its modern interpretation, the cosmological constant represents a fixed energy density permeating all of space uniformly — what is sometimes called vacuum energy or the energy of empty space.^{19, 24} Unlike matter or radiation, which dilute as the universe expands, a cosmological constant maintains a constant energy density as space grows. Its effect on the equations of general relativity is to produce a repulsive force that increases in proportion to distance, driving the accelerating expansion observed. The equation of state of this component — the ratio of its pressure to its energy density, denoted w — is exactly −1 for a pure cosmological constant.²²

The standard model of cosmology incorporating the cosmological constant is known as ΛCDM (Lambda Cold Dark Matter). It is the framework that best fits the full suite of cosmological observations: supernova distances, the cosmic microwave background, large-scale galaxy clustering, and the baryon acoustic oscillation ruler.⁷ In ΛCDM, the universe is spatially flat, 13.8 billion years old, and currently expanding at a rate of approximately 67.4 kilometers per second per megaparsec.⁷

Independent observational evidence

While the supernova measurements provided the initial discovery, the accelerating expansion and the existence of dark energy have since been independently confirmed through several entirely different observational methods.

The cosmic microwave background (CMB), the faint afterglow of radiation left over from the hot early universe, carries a detailed imprint of conditions at recombination, some 380,000 years after the Big Bang. The precise angular scale of temperature fluctuations in the CMB constrains the total energy content of the universe. The Planck satellite's all-sky CMB maps, published in a definitive analysis in 2020, found the universe to be spatially flat to within 0.2%, which requires a total energy density equal to the critical density. Since the measured matter density falls far short of this critical value, something else — dark energy — must make up the difference.⁷

Baryon acoustic oscillations (BAO) provide a third and independent line of evidence. In the hot, dense plasma of the early universe, sound waves propagated through the baryon-photon fluid until recombination, when the universe became transparent and the waves froze in place. This left a preferred clustering scale imprinted in the distribution of galaxies across the cosmos — a characteristic separation of approximately 500 million light-years that can be used as a standard ruler to measure the expansion history. BAO signals detected in galaxy surveys by the Sloan Digital Sky Survey and the 2dF Galaxy Redshift Survey in 2005 independently confirmed the geometry and energy content implied by dark energy.^{10, 11}

Large-scale structure surveys, which map the three-dimensional distribution of hundreds of millions of galaxies, also constrain dark energy through the growth rate of cosmic structure. Dark energy suppresses the formation of galaxy clusters relative to a matter-dominated universe, and the observed cluster counts and power spectrum of galaxy clustering are consistent with ΛCDM.^{14, 12}

Observational probes of dark energy and their key measurements^{1, 7, 10, 13}

Theoretical candidates for dark energy

The cosmological constant is mathematically the simplest form of dark energy, but it is not the only possibility theorists have explored. Three broad classes of models have received the most attention: the cosmological constant itself, quintessence, and phantom energy.

The cosmological constant (Λ) remains the leading candidate by parsimony. Its equation of state w = −1 is precisely constant across all of cosmic time. In this picture, dark energy is an intrinsic property of spacetime itself rather than a dynamical field, and its energy density does not change as the universe expands.¹⁹ The ΛCDM model built on this assumption fits essentially all current cosmological data within measurement uncertainties.⁷

Quintessence refers to a class of models in which dark energy is a dynamical scalar field that evolves over cosmic time.⁵ First proposed by Ratra and Peebles in 1988, quintessence models allow the equation of state parameter w to vary between −1 and 0, with w approaching −1 from above at late times to mimic the cosmological constant. A rolling scalar field would have different implications for the fate of the universe and, in some models, for the coupling between dark energy and the other fields of physics.^{5, 15} Quintessence was invoked partly as a way to address the cosmological constant problem (discussed below), since a dynamical field might naturally relax toward its minimum energy state rather than maintaining a fixed, enormous vacuum energy.¹⁶

Phantom energy describes models in which the equation of state crosses below w = −1. Such models, first analyzed by Caldwell and collaborators, have a peculiar property: the energy density of the field increases as the universe expands, because the phantom field violates the weak energy condition.¹⁷ In phantom energy models, the repulsive force grows without bound and eventually overcomes not just gravity between galaxies but the electromagnetic and nuclear forces binding atoms and nuclei together. The endpoint of this scenario is a Big Rip, in which all matter is torn apart at a finite time in the future.¹⁷

In practice, cosmologists parameterize the equation of state using the Chevallier-Polarski-Linder (CPL) form: w(a) = w₀ + w_a(1−a), where a is the cosmic scale factor. This allows a single framework to probe the full space between a pure cosmological constant (w₀ = −1, w_a = 0) and various dynamical models.²²

The energy budget of the present-day universe⁷

The cosmological constant problem

The most profound theoretical difficulty surrounding dark energy is not explaining its existence but explaining its magnitude. Quantum field theory, the framework that successfully describes all known particles and forces, makes a natural prediction for the energy density of the vacuum: quantum fluctuations — the ceaseless creation and annihilation of virtual particle-antiparticle pairs throughout all of space — should contribute an enormous energy density to the vacuum state.⁶

When the energy scales of known physics are taken into account, quantum field theory predicts a vacuum energy density on the order of 10¹¹³ joules per cubic meter. The observed dark energy density, inferred from the accelerating expansion, is approximately 10⁻⁹ joules per cubic meter.^{6, 19} The discrepancy between these two numbers is approximately 120 orders of magnitude — a factor of 10¹²⁰. This is the largest known discrepancy between a theoretical prediction and an observation in all of physics, and it has been described by Steven Weinberg as a fundamental crisis for theoretical cosmology.⁶

One resolution that has been proposed is the anthropic argument: out of a vast landscape of possible universes with different values of Λ, only those with a small enough cosmological constant allow galaxies and stars to form, and therefore only those universes can be observed by beings like us.⁴ This reasoning, associated with the multiverse hypothesis, remains deeply controversial as it is not a prediction in the conventional scientific sense. Nevertheless, in 1987 Weinberg used the anthropic principle to predict that Λ should be nonzero but much smaller than the natural quantum field theory value, anticipating the discovery by a decade.⁶

No accepted dynamical mechanism has been found that would naturally drive the vacuum energy to precisely the observed small but nonzero value. The cosmological constant problem remains unsolved and is widely considered the most pressing open question in the interface between quantum mechanics and general relativity.^{19, 24}

DESI 2024 and hints of evolving dark energy

The Dark Energy Spectroscopic Instrument (DESI), installed at the Kitt Peak National Observatory in Arizona, is the most powerful galaxy redshift survey ever constructed. By measuring the spectra of tens of millions of galaxies and quasars, DESI can map the three-dimensional distribution of cosmic structure across a large fraction of the observable universe and measure the BAO standard ruler at multiple epochs in cosmic history.¹³

In April 2024, the DESI collaboration published its first-year cosmological results based on baryon acoustic oscillation measurements across six distinct redshift bins, probing cosmic history from z = 0.1 to z = 4.2.¹³ When the DESI BAO measurements were combined with CMB data from the Planck satellite and supernova distance data from the Union3 and DES Year 5 compilations, the best-fit cosmological model showed a mild but statistically notable preference for w₀ > −1 and w_a < 0 in the CPL parameterization — meaning dark energy may have been stronger in the past and is becoming less repulsive over time. The statistical tension with a pure cosmological constant (Λ) reached a significance of approximately 2.5σ to 3.9σ depending on which supernova compilation was used.¹³

Such a significance level is intriguing but falls below the 5σ threshold conventionally required to claim a discovery in particle physics and cosmology. The result could reflect systematic errors in one of the datasets, particularly the supernova samples, rather than genuine physics beyond ΛCDM. The DESI collaboration itself was careful to note that the evidence is insufficient to conclude that the cosmological constant is ruled out, and that future data releases — DESI is expected to run for five years and will eventually survey more than 40 million galaxies — will clarify whether the hint strengthens or fades.¹³ Nevertheless, the result galvanized discussion within the cosmological community and underscored the importance of pursuing multiple independent probes of dark energy with next-generation facilities.

The fate of the universe

The long-term fate of the universe depends critically on the true nature of dark energy. Three broad scenarios are possible, distinguished by whether the equation of state parameter w equals, is greater than, or is less than −1.¹⁸

If dark energy is a pure cosmological constant with w = −1, the universe will expand forever at an accelerating rate while its energy density remains constant. Galaxies will progressively recede beyond each other's event horizons; the Local Group (the Milky Way and Andromeda, along with their satellite galaxies) will eventually merge into a single elliptical galaxy surrounded by an increasingly empty, cold, and dark cosmos. Over timescales of 10¹⁰⁰ years and beyond, even black holes will evaporate through Hawking radiation, and the universe will approach a state of maximum entropy sometimes called the Big Freeze or heat death.¹⁸

If dark energy is a quintessence field with w > −1, the expansion accelerates less vigorously. In some quintessence models, the field eventually approaches a cosmological constant, giving an outcome similar to the Big Freeze. In others, the field decays and dark energy fades, and the expansion might eventually cease and reverse — leading to a Big Crunch in which the universe collapses back into a singularity, though this requires a sustained deviation from w = −1 that current data do not favor.¹⁸

If dark energy is phantom energy with w < −1, the energy density increases with time and the repulsive force grows without limit. Gravity, electromagnetism, and finally the strong nuclear force are successively overwhelmed, and all matter — from galaxy clusters down to individual atoms — is torn apart in a Big Rip. Caldwell et al. showed that for a constant equation of state w = −1.5, the Big Rip would occur approximately 22 billion years in the future.¹⁷ Current observational constraints on w are consistent with −1 to within roughly 5%, leaving phantom scenarios alive but disfavored relative to the cosmological constant.^{7, 16}

Future missions including the Euclid space telescope (launched 2023), the Nancy Grace Roman Space Telescope (planned for the late 2020s), and the Vera C. Rubin Observatory will measure dark energy's equation of state to unprecedented precision, potentially distinguishing between these scenarios within the coming decade.