vectorize function

Returns a vector containing the upper triangle of a matrix, without the diagonal.