Big props aren't going to be a problem except if they run into the frame or each other (or your motors don't have the torque to accelerate/decelerate them quickly). Typically a 5" quad means it uses 5" props (you have 6" here the 6045 means 6.0 inch diameter, 4.5 inch pitch) I personally use 5045 5.0 inch with 4.5 inch pitch.
Main issue here is likely that you are just sending equal speed signal to all the motors but this isn't resulting in equal thrust. Basically there are a few issues that will cause the thrust to need to come out differently really (needs stabilization) to work at all really. If you could have simultaneous start (all motors start exactly at the same time) and all exactly the same specs on the motors then you could in theory just apply the same speed to all of them and get the same thrust out with the same props (also assuming props are perfect and balanced). Since none of those perfections exist in reality you need to have stabilization in place to some degree for it to fly at all really, the motors will start at slightly different times (calibration of ESCs will help reduce the difference but there will still be different static friction different windings, different magents etc. etc. that lead to them not actually starting up or accelerating in sync completely). Since one prop/motor/esc combo is going to be faster and stronger than the rest it will automatically end up tipping the quad from one corner or another and to compensate you need to be reading what is level and adjust the throttle signals to each ESC accordingly in order to achieve any sort of stable flight (even just up and down).
---
Edit additionally even if you had perfectly equal thrust you'd need the quad itself to be perfectly balanced; center of gravity (CG) right at center of quad and thrust at center of quad if you have thrust away from CG the quad will pivot around the CG.
Some interesting stuff about how quads balance in video form here too, Tom Stanton does a lot of informative projects some involving quads: