An important characteristic of geophysically turbulent flows is the transfer of energy between scales. Balanced flows pass energy from smaller to larger scales as part of the well‐known upscale cascade while submesoscale and smaller scale flows can transfer energy eventually to smaller, dissipative scales. Much effort has been put into quantifying these transfers, but a complicating factor in realistic settings is that the underlying flows are often strongly spatially heterogeneous and anisotropic. Furthermore, the flows may be embedded in irregularly shaped domains that can be multiply connected. As a result, straightforward approaches like computing Fourier spatial spectra of nonlinear terms suffer from a number of conceptual issues. In this paper, we develop a method to compute cross‐scale energy transfers in general settings, allowing for arbitrary flow structure, anisotropy and inhomogeneity. We employ a Green's function approach to the kinwetic energy equation to relate kinetic energy at a point to its Lagrangian history. A spatial filtering of the resulting equation naturally decomposes kinetic energy into length scale dependent contributions and describes how the transfer of energy between those scales takes place. The method is applied to a doubly periodic simulation of vortex merger, resulting in the demonstration of the expected upscale energy cascade. Somewhat novel results are that the energy transfers are dominated by pressure work, rather than kinetic energy exchange, and dissipation is a noticeable influence on the larger scale energy budgets. We also describe, but do not employ here, a technique for developing filters to use in complex domains.