Who is "they" in the poem, They Flee From Me?

The "they" of the poem is left ambiguous, but the term denotes a group of women who, according to the speaker, used to show him affection.