I'd lose the fleece altogether and just go with a long sleeved base layer that has a zip (usefull for venting). Then when you get a little cold, or when you stop, throw on a light softshell. Keep a waterproof shell (top and bottom) in your bag incase it rains heavy. If your camping, then the NF fleece could do you good at night around camp or when your sleeping.
I tend to go with a long sleeved base layer, then put on my softshell, and if it gets extra windy or cold then I throw on a light wind shirt. 80% of the time this will keep out most elements. If i get too hot then sometimes all thats needed is the wind shirt. I also carry a light belay jacket for when I'm stopped but its not an essential !
The reason most people remain drier in a softshell than they would in their hardshell is that it breathes a lot more so you dont end up wet with your own sweat that the hardshell otherwise traps in.
Hope this helps