tryNaN function

try an expression, and return NaN on failure