Is it ok for the tips of the arms where the motors are to vibrate?
It looks like a lot of people are having this same issue with 15 inch props too.
It could be we were wrong when we said that as long as the props didn't touch it would be ok. Since you're props and motors are balanced the only thing left is the prop clearance issue. My guess, not a proven fact, is there is a shear force in action at the tips of the counter rotating props. Sort of like how a torque converter in an automatic transmission works. That is likely the cause of the arm tips vibrating and could be the cause of the arm oscillations. You have the front and rear motors changing speeds constantly while rotating in different directions. The more I think about it the more I'm convinced that we, or at least, I was wrong to think there wouldn't be an issue with the props that close.
I used 15X5.5 props on mine with no problem but I had a few inches clearance. You could try some 11 or 12" just to see if the tip vibrations go away. I think you are spot on about the mounts being the weak link and causing the arm oscillations. I have seen quite a variety of aluminum mounts listed online. Maybe there is a design that would work for you.
If the smaller props eliminate the tip vibration then I would suggest that you seriously consider extending the rear arms. You could do it by using arm extensions like the ones from Phoenix Flight Gear or by replacing the arms with ones the right length. You are using quality materials, motors and props so it is likely a design issue rather than components.
I don't have a background in fluid dynamics, but from building and racing cars and motorcycles way back when we learned a lot about turbulence and shear forces.The mounts are an obvious problem. The shear at the props tips is just my guess. I think you could fix this for far less than the new frame you are looking at.
One more thing. Like you, I no longer buy the cheap stuff. Since using only the best that I can afford (and checking the prop balance) I discovered that it is no longer necessary to use isolation materials for mounting cameras or any other equipment for that matter. Again. I think your current problem is solvable by a design adjustment. It would be rare to work perfect the first time.