How do word representations geometrically encode linguistic abstractions like part of speech? We find that many linguistic features are encoded in low-dimensional subspaces of contextual word representation spaces, and these subspaces can causally influence model predictions.