Calculation of the geodetic distance on Earth

Understanding the Calculation of the Geodetic Distance on Earth

Geodetic distance calculation determines the shortest path between two points on Earth’s surface. This article explores precise mathematical methods and practical applications.

Discover comprehensive formulas, variable explanations, and real-world examples for accurate geodetic distance computation worldwide.

  • ¡Hola! ¿En qué cálculo, conversión o pregunta puedo ayudarte?
Pensando ...
  • Calculate the geodetic distance between New York City and London using the Vincenty formula.
  • Determine the shortest path between two GPS coordinates in Tokyo and Sydney.
  • Compute the geodetic distance for a flight route from Paris to Cairo considering Earth’s ellipsoid shape.
  • Find the distance between two points near the equator using the Haversine formula.

Common Geodetic Distance Values for Reference

Location PairLatitude 1 (°)Longitude 1 (°)Latitude 2 (°)Longitude 2 (°)Approximate Geodetic Distance (km)Method Used
New York City – London40.7128 N74.0060 W51.5074 N0.1278 W5,570Vincenty
Tokyo – Sydney35.6895 N139.6917 E33.8688 S151.2093 E7,820Vincenty
Paris – Cairo48.8566 N2.3522 E30.0444 N31.2357 E3,200Vincenty
Equator Points (0°, 0°) – (0°, 90°)0.00000.00000.000090.000010,007Haversine
San Francisco – Honolulu37.7749 N122.4194 W21.3069 N157.8583 W3,850Vincenty
Moscow – Beijing55.7558 N37.6173 E39.9042 N116.4074 E5,790Vincenty
Buenos Aires – Santiago34.6037 S58.3816 W33.4489 S70.6693 W1,140Vincenty
Cape Town – Nairobi33.9249 S18.4241 E1.2921 S36.8219 E4,850Vincenty
London – Cairo51.5074 N0.1278 W30.0444 N31.2357 E3,480Vincenty
Los Angeles – Mexico City34.0522 N118.2437 W19.4326 N99.1332 W2,500Vincenty

Fundamental Formulas for Geodetic Distance Calculation

Calculating the geodetic distance involves understanding Earth’s shape, which is best approximated by an oblate spheroid (ellipsoid). The most accurate methods use ellipsoidal models rather than spherical approximations.

1. Haversine Formula (Spherical Approximation)

The Haversine formula calculates the great-circle distance between two points on a sphere given their longitudes and latitudes.

distance = 2 * R * arcsin(√(sin²((Δφ)/2) + cos(φ₁) * cos(φ₂) * sin²((Δλ)/2)))
  • distance: Geodetic distance between two points (meters or kilometers)
  • R: Radius of the Earth (mean radius ≈ 6,371 km)
  • φ₁, φ₂: Latitude of point 1 and point 2 in radians
  • Δφ: Difference in latitude (φ₂ – φ₁) in radians
  • Δλ: Difference in longitude (λ₂ – λ₁) in radians

Common values: Earth’s mean radius R = 6,371 km. This formula assumes a perfect sphere, which introduces errors up to 0.5% for long distances.

2. Vincenty’s Formulae (Ellipsoidal Model)

Vincenty’s formulae provide an iterative method to calculate geodesic distances on an ellipsoid, accounting for Earth’s flattening.

a = major semi-axis (equatorial radius)
b = minor semi-axis (polar radius)
f = flattening = (a – b) / a
U₁ = arctan((1 – f) * tan(φ₁))
U₂ = arctan((1 – f) * tan(φ₂))
L = λ₂ – λ₁
λ = L (initial value)
Iterate λ until convergence:
sinσ = √((cosU₂ * sinλ)² + (cosU₁ * sinU₂ – sinU₁ * cosU₂ * cosλ)²)
cosσ = sinU₁ * sinU₂ + cosU₁ * cosU₂ * cosλ
σ = arctan2(sinσ, cosσ)
sinα = (cosU₁ * cosU₂ * sinλ) / sinσ
cos²α = 1 – sin²α
cos2σm = cosσ – (2 * sinU₁ * sinU₂) / cos²α
C = (f / 16) * cos²α * (4 + f * (4 – 3 * cos²α))
λ = L + (1 – C) * f * sinα * (σ + C * sinσ * (cos2σm + C * cosσ * (-1 + 2 * cos2σm²)))
Distance = b * A * (σ – Δσ)
  • a: Equatorial radius (meters), commonly 6,378,137 m (WGS-84)
  • b: Polar radius (meters), commonly 6,356,752.3142 m (WGS-84)
  • f: Flattening factor, approximately 1/298.257223563 (WGS-84)
  • φ₁, φ₂: Latitudes of points 1 and 2 in radians
  • λ₁, λ₂: Longitudes of points 1 and 2 in radians
  • U₁, U₂: Reduced latitudes
  • σ: Angular distance on the sphere
  • α: Azimuth angle
  • A, Δσ: Auxiliary variables for distance correction

Vincenty’s method converges rapidly and is accurate to within 0.5 mm on Earth’s surface, making it the preferred method for geodetic calculations.

3. Spherical Law of Cosines

Another spherical approximation, simpler but less accurate for small distances.

distance = R * arccos(sin(φ₁) * sin(φ₂) + cos(φ₁) * cos(φ₂) * cos(Δλ))
  • Variables as defined in the Haversine formula.

This formula is computationally efficient but less stable for antipodal points or very small distances.

4. ECEF (Earth-Centered, Earth-Fixed) Cartesian Distance

Converts geodetic coordinates to Cartesian coordinates, then calculates Euclidean distance.

X = (N(φ) + h) * cos(φ) * cos(λ)
Y = (N(φ) + h) * cos(φ) * sin(λ)
Z = ((1 – e²) * N(φ) + h) * sin(φ)
Distance = √((X₂ – X₁)² + (Y₂ – Y₁)² + (Z₂ – Z₁)²)
  • φ: Latitude in radians
  • λ: Longitude in radians
  • h: Height above ellipsoid (meters)
  • N(φ): Radius of curvature in the prime vertical = a / √(1 – e² * sin²(φ))
  • : Square of eccentricity = 2f – f²

This method is useful for 3D distance calculations including altitude differences but does not represent the shortest surface path.

Detailed Explanation of Variables and Their Typical Values

  • Latitude (φ): Angular distance north or south of the equator, ranging from -90° to +90°. Converted to radians for calculations.
  • Longitude (λ): Angular distance east or west of the Prime Meridian, ranging from -180° to +180°. Converted to radians.
  • Earth’s Equatorial Radius (a): Approximately 6,378,137 meters (WGS-84 standard).
  • Earth’s Polar Radius (b): Approximately 6,356,752.3142 meters (WGS-84 standard).
  • Flattening (f): Ratio describing Earth’s oblateness, approximately 1/298.257223563.
  • Radius of Earth (R): Mean radius used in spherical models, approximately 6,371 km.
  • Height (h): Elevation above the ellipsoid, often zero for surface calculations.
  • Eccentricity squared (e²): Calculated as 2f – f², approximately 0.00669437999014.

Real-World Applications and Case Studies

Case Study 1: Aviation Route Planning Between New York City and London

Accurate geodetic distance calculation is critical for flight planning, fuel estimation, and navigation. Consider the coordinates:

  • New York City: 40.7128° N, 74.0060° W
  • London: 51.5074° N, 0.1278° W

Using Vincenty’s formula, the iterative process begins by converting degrees to radians:

  • φ₁ = 0.710572 radians
  • λ₁ = -1.291648 radians
  • φ₂ = 0.898973 radians
  • λ₂ = -0.002230 radians

Calculate flattening and reduced latitudes:

  • f = 1 / 298.257223563 ≈ 0.0033528
  • U₁ = arctan((1 – f) * tan(φ₁)) ≈ 0.7090 radians
  • U₂ = arctan((1 – f) * tan(φ₂)) ≈ 0.8974 radians

Set initial λ = L = λ₂ – λ₁ = 1.289418 radians. Iterate λ using Vincenty’s equations until convergence (usually within 5-10 iterations).

After convergence, compute σ, sinα, cos²α, cos2σm, C, and finally the distance:

Distance ≈ 5,570 km

This value aligns with commercial flight distances and is used for fuel load calculations and estimated flight time.

Case Study 2: Geodetic Distance Between Tokyo and Sydney for Submarine Cable Routing

Submarine cable projects require precise geodetic distances to estimate cable length and cost. Coordinates:

  • Tokyo: 35.6895° N, 139.6917° E
  • Sydney: 33.8688° S, 151.2093° E

Convert to radians:

  • φ₁ = 0.6229 radians
  • λ₁ = 2.4381 radians
  • φ₂ = -0.5911 radians
  • λ₂ = 2.6391 radians

Calculate flattening and reduced latitudes:

  • f = 0.0033528
  • U₁ = arctan((1 – f) * tan(φ₁)) ≈ 0.6210 radians
  • U₂ = arctan((1 – f) * tan(φ₂)) ≈ -0.5890 radians

Initial λ = L = λ₂ – λ₁ = 0.201 radians. Iterate λ until convergence.

After convergence, calculate σ, sinα, cos²α, cos2σm, C, and distance:

Distance ≈ 7,820 km

This precise distance informs cable length, signal repeaters placement, and cost estimation.

Additional Considerations for Geodetic Distance Calculations

  • Earth’s Shape Variability: Earth is not a perfect ellipsoid; local geoid variations affect precise measurements.
  • Altitude Effects: Including height above sea level can refine distance calculations, especially for aviation and satellite applications.
  • Computational Efficiency: For large datasets, approximations like Haversine may be preferred despite lower accuracy.
  • Software Libraries: Libraries such as GeographicLib and PROJ implement Vincenty and other geodesic algorithms efficiently.
  • Coordinate Systems: Ensure consistent datum usage (e.g., WGS-84) to avoid systematic errors.