How I Research Company Reviews — My AI Report Methodology

How I Research Company Reviews — Full Methodology

I publish AI-assisted research reports on companies in the debt relief, credit repair, and consumer finance space. These reports compile publicly available data — corporate filings, court records, government complaints, consumer reviews, and website compliance — so consumers can make informed decisions. Here is exactly how every report gets built, from start to finish.

Why I Built This Process

When consumers search “Is [Company] legit?” they find a mix of affiliate review sites (paid to send leads), AI-generated content (unsourced assertions echoing each other), and genuine reviews that may describe a phone call rather than actual program results.

I built this research process to produce something different: a structured, sourced, verifiable compilation of what the public record actually shows about a company — the good and the bad, weighted equally.

The Research Team: Eight Specialized AI Agents

Each report starts with up to eight specialized AI research agents working in parallel. Each agent has a defined scope and a specific set of databases to search. They do not communicate with each other — they report independently to avoid groupthink or confirmation bias.

Agent 1: Corporate Records Investigator

Traces the company’s legal history through state corporate filings. This agent follows chains of evidence: find the legal entity, find the principal’s name, search that name for other entities, search those entities for shared addresses. What emerges is a corporate genealogy — who owns what, who is connected to whom, and how the company has changed over time.

Critical rule: Every corporate connection claim must be verified through state filings — not third-party blogs. If a review site says “Company A is owned by Company B,” this agent checks whether that claim is supported by actual corporate records. Claims that cannot be verified are flagged as unconfirmed.

Agent 2: Court and Enforcement Analyst

Searches federal court records and government enforcement databases for lawsuits, consent orders, and regulatory actions involving the company and its principals.

This agent also searches principal names individually — an executive’s involvement in a previous enforcement action at a different company is relevant context for consumers.

Agent 3: Consumer Review Analyst

Collects and analyzes consumer reviews across all major platforms. Agent 3 does not just read the most recent reviews — it pages through multiple pages on each platform to collect reviews spanning different time periods, typically 30 to 50 reviews per platform. This avoids recency bias and captures both older and recent experiences.

Platforms searched: Trustpilot, Better Business Bureau, Google Reviews, BestCompany, and PissedConsumer.

Agent 3 classifies every collected review into one of three categories: Call/Intake, Program Performance, or Ambiguous. It also writes the raw review text (without its classifications) to a separate file for Agent 8 to classify independently. The full classification methodology is described below.

Agent 4: Web and Background Investigator

Researches leadership backgrounds, marketing practices, and third-party claims about the company. This agent is specifically tasked with finding claims made about the company on other websites — and checking whether those claims are accurate.

Why the Facebook Ad Library matters: It reveals which companies are actively advertising and whether third parties are running ads using the company’s name. This can uncover lead generation networks, celebrity endorsement campaigns, and multi-brand operations that consumers would not otherwise discover.

Agent 5: Website Compliance Analyst

Reads the company’s entire website with a regulatory compliance lens. For debt settlement companies, this means checking for all eight FTC Telemarketing Sales Rule (TSR) required disclosures. For other company types, the checklist adapts — FDCPA for debt collectors, TILA for lenders, CROA for credit repair companies.

This agent documents what the company discloses and what it does not. Disclosures that are present get credit. Disclosures that are absent are noted as gaps — not as accusations.

Agent 6: Complaint and Response Analyst

Analyzes formal complaint data from the CFPB Consumer Complaint Database and the Better Business Bureau. But the complaint count is only half the picture — this agent also reads how the company responds to complaints.

Every sampled company response is classified by tone: Resolution-Oriented (reports use “Tried to Fix It”), Procedural/Template (“Generic / Copy-Paste Reply”), Defensive (“Defensive / Blamed Customer”), or Non-Responsive (“No Response”). A company that consistently offers specific resolutions to unhappy customers tells a very different story than one that sends template responses.

Agent 7: Nonprofit Financial Analyst (Conditional)

This agent deploys only when the company under review is a 501(c)(3) nonprofit organization. It retrieves and analyzes the organization’s IRS Form 990 tax filings via the ProPublica Nonprofit Explorer API, extracting executive compensation, program service revenue, total assets, and how the organization allocates funds between program services and administrative overhead.

For-profit companies skip this agent entirely. When it does deploy, its findings feed into dedicated report sections on financial transparency and executive compensation that are unique to nonprofit reviews.

Agent 8: Independent Review Classifier

This agent exists for one purpose: to independently verify Agent 3’s review classifications. It receives only the raw review text — no access to Agent 3’s decisions — and classifies each review using the same fixed criteria. The two sets of classifications are then compared to measure agreement.

This is the inter-rater reliability step described in detail below.

Data Sources

Every report draws from publicly available databases. No private records, leaked data, or insider information is used.

Connection Analysis: How I Trace Corporate Relationships

Many companies in the debt relief space share principals, addresses, or corporate histories with other entities. A consumer signing up with “Company A” may not realize it was created by the same people who ran “Company B” — which may have enforcement actions against it.

I use a tiered evidence framework to classify every connection claim:

Every connection in every report is labeled with its evidence tier. Readers can see exactly how strong the evidence is and judge for themselves.

Review Analysis: Separating Calls From Results

Why Review Ratings Can Be Misleading

Most debt relief companies have thousands of online reviews with high ratings — 4.5 stars or above on Trustpilot, Google, and similar platforms. Those numbers look great until you read the actual reviews.

Debt settlement programs typically last 24 to 48 months. During that time, consumers pay fees, their credit scores drop, and creditors may sue. But many companies send review requests within days of enrollment — at the moment of peak hope, before any of those consequences arrive.

The result: a company’s published rating might be 4.7 stars, but if 75% of those reviews describe the phone call and only 25% describe the actual program, consumers are making decisions based on incomplete data.

My Classification System

Every review in my sample gets classified into one of three categories using fixed criteria defined before the data is collected:

Ambiguous reviews — those that do not clearly fit either category — are classified separately and excluded from the adjusted rating calculation. They are still counted and reported.

How Reviews Are Collected

Reviews are not cherry-picked. For each platform where a company has reviews, Agent 3 reads through multiple pages to collect 30 to 50 reviews spanning different time periods — not just the first page of most recent reviews. If a platform has fewer than 50 reviews total, all reviews are collected.

Why Not Larger Samples?

The statistically ideal sample for a company with 16,500 Trustpilot reviews would be 376 reviews (Cochran’s formula: 95% confidence level, ±5% margin of error). That math is straightforward. The review platforms, however, impose access restrictions.

Rather than claim sample sizes that cannot be reliably collected, this process is transparent about what review platforms allow. The reviews that are accessible are collected and analyzed. The number collected, the total available, and any access limitations are documented in every report.

What This Means for Precision

Every report discloses: the number of reviews collected per platform, the total reviews available, any access limitations encountered, and the resulting confidence intervals. Readers can evaluate the statistical strength for themselves.

The dual-classifier approach compensates. While a smaller sample means wider margins on the rating estimate, the inter-rater reliability measurement (Cohen’s kappa) is independent of sample size — it measures whether the classification criteria are being applied consistently. A kappa of 0.75 on 40 reviews is just as meaningful as a kappa of 0.75 on 376 reviews. The classification system’s objectivity is verified regardless of how many reviews are collected.

Independent Verification

Dual AI Review Classification

Classification is subjective. Two reviewers reading the same text might disagree about whether it describes a phone call or a program outcome. To measure and account for this, I use two independent classifiers.

Cohen’s Kappa

I measure agreement between the two classifiers using Cohen’s kappa (κ), a statistical measure that accounts for agreement that would happen by chance:

Kappa is computed for every report. The full kappa score and contingency matrix are available in the research files. Reports themselves use plain language — “two AI reviewers independently sorted every review” — with a link back to this page for readers who want the statistical details. Low kappa is not a failure — it is data. It means the company’s reviews genuinely mix call and program content, which is itself useful information for consumers.

Resolving Disagreements

When the two classifiers disagree on a review:

• One says ambiguous, the other commits to a category: I use the non-ambiguous classification (the classifier who committed had a reason)
• One says call/intake, the other says program performance: The review is classified as ambiguous (genuine disagreement — I do not force a category)
• Both agree: That classification stands

The Adjusted Rating

After classification and disagreement resolution, I compute two separate ratings for each platform:

Program Performance Rating (called “Results-Only Rating” in reports) = average star rating of reviews classified as describing actual program outcomes.

Call/Intake Rating (described as “reviews about the sales call” in reports) = average star rating of reviews classified as describing the initial call experience.

The gap between the published rating and the Results-Only Rating tells consumers how much of the company’s rating comes from program results versus sales call experiences. Reports present these numbers in plain language; the technical methodology is documented here.

Confidence Intervals

Every adjusted rating includes a 95% confidence interval:

If only 8 program reviews exist out of 40-50 collected, the confidence interval will be wide (possibly ±1+ stars). I report that honestly. A wide CI means: “There are not enough program-outcome reviews to be precise about this company’s actual service quality.” That is itself a finding — and it often reveals that the vast majority of a company’s reviews describe the sales call, not the program.

Dual AI Cross-Check (GPT Review)

After the report is written and fact-checked, the entire document is submitted to a second AI engine (GPT, by OpenAI) for independent review. GPT checks for:

• Factual accuracy — do numbers and dates match known public records?
• Logical consistency — do sections contradict each other?
• Missing public data — is there significant information about the company not included?
• Fairness — is any section framed in a way that implies wrongdoing rather than presenting data neutrally?
• Connection verification — are corporate ownership claims backed by state filings or only by blog posts?
• Entity completeness — is every named person and company properly tagged?

GPT’s findings are triaged into three categories: Fix (real errors — must be resolved), Improve (strengthens defensibility — usually implemented), and Skip (stylistic preferences — no action needed). Every Fix item must be resolved before the report can proceed. The GPT review artifact is saved and referenced in the quality gate.

Quality Control: The 43-Point Gate

No report reaches the site until it passes every check in a 43-point quality gate. If any check fails, the issue is fixed and the entire gate runs again from the top — because a fix in one area can introduce a new issue elsewhere.

The gate runs in three phases: an automated script handles 33 mechanical and quality checks in seconds (HTML structure, source link validation, CFPB data consistency, star rating arithmetic, connection tier labels, and more). Then seven checks requiring LLM judgment are evaluated manually (source verification, corporate connection evidence, entity type profiling, state registration confirmation, name-only connection screening, GPT fix reflection, and improvement recommendations). Three additional checks are confirmed from the research phase. Every check must show PASS before the report can proceed.

Here is a representative sample of what the gate checks:

Content and Sourcing

• Liability assessment completed and at LOW risk or below
• Every source URL fetched and confirmed — no hallucinated quotes or bios
• Review categorization (call/intake vs. program performance) included with statistics
• All external links archived on the Wayback Machine and swapped in the HTML
• No email addresses in the post — all redacted, contact form link only
• Zero first-person language anywhere in the report

Verification and Cross-Checking

• Corporate connections verified via state filings, not blog posts
• No name-only connections (similar names without shared principals or addresses)
• GPT review artifact exists with all Fix items resolved
• GPT findings are non-zero (proves the review actually happened)
• GPT fixes reflected in the final post

Completeness

• Entity tag box matches every named person and company in the report
• First-mention entity linking in HTML (every named entity linked on first reference)
• Facebook Ad Library checked and documented
• Reverse address search completed for all company addresses
• State registrations verified individually (actual database searches, not assumptions)
• “Correcting the Record” section present (debunking false third-party claims)
• Complaint response analysis with tone breakdown
• Review analysis completed with dual-classifier verification
• Inter-rater reliability computed (Cohen’s kappa)
• Results-Only Rating in report with methodology link

Liability Review

Before any report is uploaded to WordPress, it goes through a liability assessment. This is not about avoiding controversy — it is about ensuring every statement in the report is defensible.

The assessment checks for:

• Defamatory language: Does any statement accuse the company of illegal activity without sourcing to a court filing or enforcement action?
• Unsourced claims: Does every factual assertion have an inline source link?
• Opinion masquerading as fact: Words like “deceptive” or “misleading” are replaced with factual observations like “not disclosed” or “absent from website”
• Unfair framing: Are positive findings (clean enforcement record, strong BBB rating, improving trends) given equal prominence to concerning findings?
• Connection evidence: Is every corporate connection labeled with its evidence tier?

Reports must reach LOW risk before upload. If the assessment is MEDIUM or higher, the report goes back for revision until every identified issue is resolved.

The “Correcting the Record” Approach

Every report includes a section that corrects false claims made about the company by third-party websites. If a blog incorrectly states that Company A is owned by Company B, and my investigation found that the actual owner is Company C per state filings — that correction goes in the report.

Report Philosophy

These reports exist to help consumers AND to encourage companies to do better. They are not adversarial. The philosophy:

• Praise what deserves praise. If a company has a clean enforcement record, strong complaint responses, or improving review trends — say so prominently.
• Frame gaps as opportunities. “Not disclosed on website” is something a company can fix tomorrow. I present it that way.
• Lead with strengths. Every section acknowledges what the company does well before noting what is absent.
• Let the data speak. Strong compliance + clean record + good responses = the data tells a positive story. Weak compliance + many complaints = the data tells that story too. I trust the reader to see it.

What This Process Does Not Do

Transparency means acknowledging limitations:

• It does not access private records. Sealed court filings, proprietary databases, private financial records, and internal company documents are not part of this analysis.
• It does not prove reviews are fake. A call/intake review is a real experience. The analysis separates review types — it does not invalidate them.
• It does not determine the “real” rating. The Program Performance Rating is one lens. Consumers who value customer service during intake should factor in those reviews too.
• AI classifiers have biases. They may over-index on certain keywords or miss context that a human reviewer would catch. The dual-classifier approach mitigates but does not eliminate this.
• Review samples are limited by platform access. Review platforms restrict how many reviews can be accessed without authentication. Samples of 30-50 reviews per platform provide directional insight but have wider margins of error than larger samples would. Important individual reviews may be missed.
• CFPB data is voluntary. Not all consumers file complaints, so CFPB numbers undercount actual issues.
• State registration searches have limits. Some state databases require CAPTCHAs or in-person access. When a state cannot be searched, the report says so.
• AI research may miss human context. Nuances from verbal communications, in-person interactions, or industry insider knowledge may not be captured.
• This is not legal, financial, or investment advice. It is a research compilation to help consumers start their own investigation.

Correction Policy

The classification criteria, sampling method, statistical calculations, and quality gate checks described on this page are fixed as of the publication date. They apply equally to every company reviewed. No company receives favorable or unfavorable treatment in how its report is researched or written.

Methodology version 1.3 — Updated February 2026. Changes from v1.2: Updated agent count from seven to eight (added conditional Agent 7 for nonprofit 990 analysis; renumbered Independent Review Classifier to Agent 8). Removed ConsumerAffairs from review platforms (not used in practice). Added PissedConsumer to data sources list. Added state registration verification process description (15+ state SOS databases searched in parallel). Updated quality gate from 24 to 43 checks reflecting automated script plus manual judgment phases. Added description of three-phase QA automation.

Frequently Asked Questions About How I Research Companies

How do I research the companies reviewed on this site?

Every report starts with up to eight specialized AI research agents working in parallel, each with a defined scope and specific databases to search. They don’t communicate with each other, specifically to avoid groupthink or one agent’s assumption contaminating another’s findings.

What sources does the research actually check?

The page names them specifically: state corporate filings (all 50 states where relevant), SEC EDGAR, OpenCorporates, FDIC BankFind, NMLS Consumer Access, CourtListener/RECAP and PACER for court records, the CFPB complaint database, BBB profiles, Trustpilot and Google reviews, and IRS Form 990 filings (via ProPublica) for nonprofits.

Why don’t I just tell you whether a company is good or bad?

Because facts change — complaint counts go up, leadership changes, and a company with a clean record last year may not have one today. Instead of a static opinion, the reports link you directly to the live public sources so you’re seeing current data, not a stale snapshot from whenever the report was written.

How is this different from other “company review” sites?

Two problems I built this to avoid: affiliate review sites that get paid to send leads, and AI-generated content that echoes unsourced assertions from other sites. My process requires every corporate-connection claim to be verified through actual state filings — claims that can’t be verified get flagged as unconfirmed rather than stated as fact.

How thorough is the process, in numbers?

Per the page: 8 AI research agents, 15+ public databases searched, 2 AI engines used to cross-check findings, and 43 quality-gate checks before a report is finished.

Steve Rhode The Get Out of Debt Guy | Consumer Debt Expert

Consumer debt expert & investigative writer. Personal bankruptcy survivor (1990). Washington Post award-winning author. Exposing debt scams since 1994.

See Full Bio

How to Get out of Debt Money Troubles Money Personalities Budgeting Behavioral Economics Motivation Inspiration Money Management The History of Credit and Debt Getting Out of Debt Get Out of Debt Debt Settlement Credit Counseling Loan & Debt Relief Scams Debt Relief Industry Practices Exposing Deceptive Financial Services Helping Consumers Navigate Financial Hardship Consumer Advocacy Consumer Debt Expert Consumer Protection Debt Relief Scam Student Loan Bankruptcy Debt Validation Debt and Mental Health Student Loans Debt Consolidation Debt in the Military Bankruptcy Consumer Debt Credit Card Debt Debt Relief Scams Personal Finance YMYL Financial Content Chapter 7 Bankruptcy Chapter 13 Bankruptcy Medical Debt Student Loan Debt Consumer Bankruptcy Financial Fraud Exposure Collections Investigative Writing Fraud investigation True Crime Research