The federal law (FAA) requires that you keep your craft under 400 ft from the ground, fly it line of sight, not within I think 5 miles of an airport (even a small municipal airstrip), not fly over people or in populated areas. The "mileage" in your area may vary -- what is enforced.
Now for the real-world flight guidelines:
To minimize the butt-hurt, it's a good idea to stay 75 feet or higher above anyplace people might be. If they don't see it or hear it, they aren't going to get butt-hurt over it.
If you're going to fly over your neighborhood, do it at 100 feet or above, and make one pass count. Don't swoop over your neighbor's house over and over and over! Don't be "that guy".
Even in a wilderness or hiking area, people will generally not pay much attention to a quad buzzing overhead once, or maybe twice, but if it becomes irritating, they will seek to share their irritation with you.
If you're flying anything larger than a 350, stay up in the air -- don't "buzz" people. Small soft props will make people mad, but large firm props will remove body parts.
If you're going to scope out the highway/freeway with your FPV quad, don't do it directly above the roadway -- do it along one side or the other, over a frontage road, so if your quad drops out of the sky, it won't make cable news.
Snooping in apartment windows is right out. Flying over a beach with people on it will be unpopular even if it's legal. Get a good camera and stay up high.
Don't fly in an urban area ever, for any reason, at any time of the day or night. (see: don't make cable news)
Don't pilot your quad from the passenger seat of a vehicle in motion. (goes double for the driver's seat of a vehicle in motion)
Don't drop anything heavier than a ping pong ball from your quad on anyone not in on the game.
Don't drop anything from your quad that can be mistaken for a bodily substance regardless of weight. (Don't drop an actual bodily substance either)
Do not give pets, lab animals or livestock "a ride" on or under your quad. (to clarify here, your neighbor's cat qualifies as a pet)
The bottom line on real-world guidelines is: DO NOT BE OBNOXIOUS. and generally, if the only person who knows your quad is in the air is YOU, everything is going to go a whole lot smoother
