Roundtrip matrix method for calculating the leaky resonant modes of open nanophotonic structures