I reckon you need to give it a go a let us know how it goes, as I have the same question but don't want to be the guinea pig
I recall sleeping in a survivial bag when I was in my teens with only a mat in summer and I think I coped ok back then. Much lower levels mind you and alcohol may have played a part in the good nights sleep
My thinking, like mentioned, would be to include an insulated jacket into the mix. A synthetic one most likely for the obvious damp concerns. No way would I try it in anything other than a decent summer's night but weather permitting, a decent jacket and mat I think it might just be doable. Especially if you are a hot sleeper like I am.
I like to travel fast and light. I already take the jacket with me on most occasions so I'm only adding 500g (???) for a bivvy bag. That sounds great compred to hauling a tent and sleeping bag around.
Now to convince my other half its a good idea!