You're having a major case of the "whoosh". In the context of this conversation, if you are self-employed and/or own your own company, then "the man" is the government, your clients, possibly even your suppliers.