Close meta connection socket after cleaning up event structures.

Epoll doesn't like it when an already closed filedescriptor is being
removed, so we defer closing the socket until after all else is cleaned
up.