The field of relativistic astrophysics has witnessed a major revolution with the development of increasingly more sensitive telescopes and gravitational wave detectors on Earth and in space. An outstanding question is what can be learned from the observed data. In this talk, we report recent progresses on two inverse problems of reconstructing spacetime structures. The first problem is the recovery of initial status of the universe from the Cosmic Microwave Background. Mathematically, the heart of the problem is an integral transform in Lorentzian geometry, called the light ray transform. We discuss its injectivity, stability and connections to wave equations and kinetic theory. The second problem is the recovery of black hole spacetimes from gravitational wave signals observed by LIGO. In particular, we show how to "hear” the shape of black holes by using the characteristic frequencies (or quasi-normal modes) extracted from the black hole ring-down.