To be fair the midges will eat you alive on campsites as well
Finding a nice spot next to a major attraction will be hard as those places are always busy with tourists. Camping in Glen Brittle (Fairy Pools) outside of the campsite is strongly discouraged (there are signs everywhere - I don't know the legal basis for it but I have never seen a tent down there).
All that said, I'd try it, but always keep in mind where the campsites are and understand that, on the day, the campsite might actually be the best choice and just make a decision on the day.