In mid-May, Novartis went from a “best places to work” list to a black list: Companies No Sane Woman Would Work For. The company’s fall from grace came by way of a class-action discrimination lawsuit it lost — and a $250 million fine it subsequently gained.  The jury found that Novartis paid female salespeople less than male reps and had demoted some of them when they were pregnant. As part of its defense, Novartis paraded its ten-year tenure on the Working Mother “100 Best Companies for Working Mothers” list.

How could they? No, I don’t mean Novartis. I mean how could Working Mother, Diversity Inc., and others keep a company that’s mired in a class-action lawsuit on their “best” lists?

As one who is in charge of compiling such lists, I’m about to tell you what they won’t. Then, in my follow-up post on Wednesday, I’ll give you a brand-new kind of best list: “Are ‘Best Places to Work’ Lists for Real? 5 Ways to Tell.”

First, a little prologue about my past. Back in 1998, I was asked by Working Woman magazine to come up with a methodology to rank the Fortune 500 according to how well women fared in their executive ranks. Despite the fact that I’m a business journalist, I had a math epiphany: I came up with an equity index that enabled me to make apples-to-apples comparisons about how well women did at all kinds of companies, from cosmetics to construction.

Being of an entrepreneurial bent, I realized I had the makings of a company. I managed the Working Woman list for six project cycles and translated the methodology for women’s associations for various industries, including cable television and accounting. Along the way, I’ve learned quite a bit about how lists are put together. Pull on your waders, because it’s going to get deep.

Dirty Secret Number One:  It’s really easy!

Many list-compilers persuade companies to apply by emphasizing how easy it is. One company claims it takes only 20 minutes to fill out the qualifying forms. They promise free publicity for little effort. Who doesn’t love free and easy?

It’s easy to check off every box on a laundry list of programs. Flexwork? Check! Telecommuting? Check! Free cake on your birthday? Check! Commitment from the CEO to diversity initiatives? Double check!

Very few list-makers ask for statistical evidence that programs are actually used by significant numbers of employees or that the programs actually drive business results. Providing evidence of effectiveness is always inconvenient and often hard. Asking for hard things will discourage companies from participating. That’s why most list makers don’t ask — and don’t tell you that they don’t ask — for proof.

Dirty Secret Number Two:  List-makers believe everything they’re told.

Well, if it is easy to participate, surely there is follow-up to ensure that those easily listed programs correlate with greater numbers of women in management…right?

Sorry to disappoint. Nearly all list-makers take companies’ self-reported answers at face value. The few that ask for statistics rarely, if ever, check the math. Employee opinions might be collected via brief (and easy!) online surveys, but rarely in free-ranging interviews that might reveal the true workplace culture.

This explains why it was so easy for Novartis to stay on high-profile lists. The class-action suit was widely reported and on the public record — hello, it was playing out in court! –  but tying it back to listworthiness would have taken independent fact-checking. My guess is that Novartis did not mention its pending suit on its list applications, which means that the list-compilers put Novartis on their lists solely on the basis of what they were told, with no verification.

Dirty Secret Number Three:  It’s free — but there is a cost.

Real research is expensive. Catalyst, the top-notch women-in-business think tank, charges a $2,500 application fee for its Catalyst Award. European firm Best Companies, which compiles “Best Places to Work” lists for publications such as London’s Sunday Times, charges an administrative fee. That’s the model my research firm uses for some projects, because it transparently states and covers the cost of research up front, for all to see. (Costs for some of our other projects are covered by association or corporate clients.)

But the allure of many American list projects is that they are easy and free. Business people ought to know better; nothing is ever truly free. Use Facebook, give up your privacy. Use LinkedIn and find your long-cultivated professional network cherry-picked by acquaintances. And sign up for an “easy and free” list, and invite pressure to buy advertising, buy consulting services, buy upgraded employee surveys, and on and on. List-compilers claim that their editorial independence means they profile excellent programs with no regard to the advertising and sponsorship revenue supplied by participating employers. Oh, Bambi, that is so not true. With lists, as in life, you get what you pay for.

Just ask Novartis, which cited its perennial status on the Working Mother list as part of its defense — and got all that free publicity.