4: Collecting Data - New School and Old School
The author refers to a graduate student's research project, which discovered patterns in customer behavior - identifying demographic characteristics (gender, age, race, education, family size, and income bracket) and their interests in various lifestyle topics - all from public sources of information (other searches by the same users, correlations to their profiles on community Web sites, etc.). His approach enabled him to collect information on a much larger number of people, in greater detail, with greater speed, and with greater (presumed) accuracy than traditional marketing research.
Old School Information
The anecdote above seems impressive, and it's noted that a lot of information has become available on the Internet that was previously unavailable - either because it was not collected, or because it was difficult to access in a usable format.
The 1966 Freedom of Information Act (FOIA) was intended to create a more transparent government by making accessible, to any citizen, any information collected or used by the federal government (with few exceptions). The author provides a few examples of the kinds of information that is available from the government, and it's quite extensive.
Naturally, the problem is bureaucracy. While citizens have a right to access this information, it's difficult to determine which office of which branch of which agency might have information of a given type, or whether the specific information you seek is available, or how to go about obtaining it. And then there's the matter of forms and procedures and red tape.
Even so, the information is available, and while it may still require some effort to retrieve it, government data is slowly becoming available online, which greatly increases its accessibility and usefulness to competitive intelligence.
Data Regarding Individuals
The author lists some of the information that is kept on file for most civilians - it's quite extensive and ranges from judicial matters (a person may be a relative of someone who was investigated but not indicted, or a witness in a civil matter), to employment matters (if a person was in the military or hold a government job of any kind), or even applied for a license or permit, their information is in the system.
Information about individual persons can be critical, depending on the role of a person. They may be a customer or decision-maker in a key transaction, a candidate for employment or promotion, an executive of a competing firm, etc. Much information can be gathered about a person through government records, "if you know where to look."
The FOIA has limits, as do many governmental agencies, and can refuse any request if it seems to be an "unwarranted invasion of personal privacy." Chiefly, this safeguards medical, financial, and personnel records, and there's ongoing debate and legislation that is meant to maintain the balance between transparency of government and privacy of the citizens.
Data Regarding Companies
The federal government collects a considerable amount of information on companies, and because privacy concerns do not apply to organizations in the same way they apply to individuals, this information is readily available.
There is a wealth of information available on any company that is publicly traded, but there are a number of other reasons information might be on file: the firm may have bid on a government contract, its products may have been inspected by one of a plethora of regulatory agencies, etc.
In some industries (food products, auto parts, pharmaceuticals, mining, etc.), there are government agencies that pay a great deal of attention to their products and manufacturing facilities. Companies that do any business overseas, especially if product is imported or exported, must deal with government agencies.
In addition to competitive intelligence, this information can be used to gain insight into the operations of competing firms - to learn from their successes, avoid making the same mistakes, or even reverse-engineer their products and processes.
The author also notes that state, county, and city governments also have broad powers and collect a "breathtaking amount of information" - though states have their own rules and requirements for the availability of their information. Some examples include: blueprints for buildings, inspection reports for facilities and equipment, OSHA inspector reports, hazmat permits, labor reports, waste disposal permits, etc.
An Example of How to Use This Information
The author spins a narrative about how this information can be put to use.
A firm was faced with a "formidable competitor" with a 40% market share, and was concerned about a secret project that was being done, even though they had no indication of what it might be. He realized it could be a smokescreen (the other firm having a good CI department and was known to use deception to mislead its competition). The only details to start with were that there was unusual capital spending on a manufacturing facility without any press releases, as well as high turnover in the executive offices, having recently sacked a couple of C-level executives over poor stock performance.
His first thoughts were that the capital expenses could be to address an OSHA or similar issue the company needed to address but didn't want a lot of coverage about, maybe even a nonprofit stirring up trouble for them. They pulled all the companies state and local regulatory findings, with a close eye on waste disposal and construction plans for their facility, and had engineers pore over it as well.
In reviewing the regulatory findings, it was found that there were areas where information was obviously missing, and some figures were misrepresented due to apparent mathematical errors (both atypical, especially for engineers), and it was clear that the company was spending a lot of cash, very fast, to address a problem - information that would eventually make its way to te public, and which would have a negative impact on the company's stock and reputation as a supplier. The author mentions only that they "exploited" the information with customers and suppliers to gain a competitive edge over them.
(EN: The case study is all very vague, and I suspect purposefully so, as there's probably a lot the author can't disclose publicly. I don't doubt the veracity of the case study, but find that the scant detail is not as illustrative as he might have assumed it would be.)