I can't answer the question of why it won't RTH.
However, your drone has features like out of range alarm, low power alarm.
If you acted on the alarm and hit the RTH before the drone was actually out of range, maybe you would not have lost camera connection. Furthermore, when your drone is out of range, your controller will also lose connection with the drone. Maybe that's why your RTH wouldn't work?
One thing that puzzles me. A drone with GPS usually has the Auto Return Home feature when it goes out of range. Do I read your post correctly that it did not auto return when out of range?
Hope I make sense to you.