It depends on what you use them for. If you want to compare the intelligence of people from different cultures, you need a culturally neutral test (whether such a thing exists is another question). For clinical diagnosis, when what you want to test is whether an individual is retarded or if there are suspicious patterns in their test results – like a large difference between their score at different subtests – I don't think it matters.

There are (or were) actually tests used by psychologists that include testing one's vocabulary. Which is knowledge, not intelligence.