ReachIQ Crawler

Crawl status

Every company URL the crawler knows about.

Top industries (crawled)

LinkedIn industry of the companies crawled so far.

Crawled companies

Costs so far

…

Balances

balances checked every 5 min

Date	Provider	Amount (USD)	Note	Added by

Service	Worker	Basis	Spent (USD)

Last 14 days, per day

Date	jina	GPT	AWS gateway	VPS	Total

Start crawling from a LinkedIn company URL

Give a LinkedIn company page. It goes to the front of the queue; the companies on its "Similar pages" and "Affiliated pages" follow right after (two levels), before the rest of the queue. Set a country and/or industry below and only companies matching those rules are followed in full: the others are walked through for their links, but get no people search or e-mails. Leave a rule empty for "any". People, decision makers and e-mails are added by the other workers as usual.

Seeds and their chains

Seed	Status	Lane	Rules	In chain	New	Crawled	Waiting	Passed through	People	Added

Campaigns

Campaigns run one after another: line them up and the next starts when the one running is full or has nothing left to draw; a campaign started while another runs joins the list. Say which companies you want (country, industry, optionally a size band) and when to stop. A campaign first takes matching companies from the reserve (parked, by industry), then picks matching companies from production that the crawler has not opened yet and starts each as its own seed, adding a fresh batch whenever the waiting queue runs low, until the number of kept companies you set is reached. Single seeds (Start from URL) keep working as before.

Campaigns

Campaign	Status	Rules	Seeds	Kept / cap	Fetched	Kept per 100	Waiting	Passed through	Started

How it works

Four workers run one after another on every new company. Companies that production already has are never crawled. LinkedIn only ever sees AWS addresses; websites and mail servers only ever see the VPS.

In plain words: what the LinkedIn crawler does

It reads public LinkedIn company pages (the ones that open without logging in) and stores what is on them: name, website, industry, size, address, followers and the 1–4 people shown on the page. No login, no person profiles, no e-mails (LinkedIn pages have none).
It moves like "up next" on YouTube. Every company page lists 10 "Similar pages". The crawler saves one company, puts its 10 similar companies in the queue, opens the next one, and so on. One seed company (TCS, 9 Sep) has led to 57,000 companies so far. It never runs out.
Three things happen on every page: (1) check first: is this company already in ReachIQ production? If yes, skip it without opening (21,000 skipped so far); (2) open the page through AWS, a different AWS IP each time, 30 s apart, at most 2,000 pages a day, so LinkedIn never sees the office; (3) save the data in the exact production field names.

One line: follow LinkedIn's "Similar pages" to find new companies, hidden behind AWS, only the ones ReachIQ does not have yet, saved in production format — about 2,000 new companies a day.

What data the crawler gives us

Company — from the LinkedIn page (production field names)

Data	Key
Company name	`company_name`
LinkedIn page URL	`company_linkedin_url`
Website	`company_website`, `original_company_website`
Industry	`company_linkedin_industry`, `industry`
Description (About us)	`company_linkedin_description`
Specialities (list)	`company_specialities`
Number of employees	`number_of_employees`
Company size (e.g. 10,001+)	`crawl.company_size`
Followers	`number_of_follower`
Founded year	`company_founded_on`
Type (public, private, nonprofit…)	`organization_type`
Headquarters	`crawl.company_headquarters`
Address, city, state, country, postal	`company_address`, `company_city`, `company_state`, `company_country`, `crawl.postal_code`
Similar / affiliated pages	`crawl.similar_pages`, `crawl.affiliated_pages`
Stage / source / dates	`sanitize_stage` 39, `data_source`, `created_at`, `updated_at`

People — 1 to 4 per company, from the "Employees" cards

Data	Key
First name, last name, full name	`first_name`, `last_name`, `full_name`
LinkedIn profile URL	`contact_linkedin_url`
Link to the company	`company_id` + company fields copied (`company_name`, `company_website`, `industry`, `company_country`…)
Where the person came from	`crawl.source` = company_page
Job title	`job_title` — empty here, filled by the decision-maker search

Added later by the other workers (not on LinkedIn)

Data	Key	Worker
Phone number	`company_phone_number`	website scan
E-mail + status	`email`, `email_verification_status`, `email_verification_updated_at`	pattern + verifier
Decision makers with title	`job_title`, `crawl.seniority`, `crawl.source` = search	search + GPT judge

1Find companies

crawler · linkedin-chain-crawler

Starts from one company's LinkedIn page. Its "Similar pages" and "Affiliated pages" give 10–15 more companies, which go into the queue.
Every link is first checked against production (primary_companies_1, 4.27M companies). Already there → skipped ("in prod, skipped"); new → queued.
Each queued company page is opened through AWS API Gateway: a different AWS IP per request, the office IP never reaches LinkedIn. 30 s between pages, max 2,000 pages a day.
From the page: name, website, industry, size, HQ / address, followers, description and the 1–4 employees shown (name + profile URL). Saved in production format → "Crawled (done)".
The new links from that page go back into the queue, so the chain keeps going.

Output: ~2,000 new companies a day

2Decision makers

linkedin-people-search · jina.ai search + GPT-6 Luna

For every new company, 3 web searches (jina.ai): "<company> CEO OR founder OR director…" in English, in the country's language, and with the website domain → 20–30 LinkedIn profile results (name, title line, snippet).
GPT-6 Luna judges each result: is this person the current CEO / founder / director of this company? Wrong company, former employees and interns are rejected. The judge runs through the OpenAI Batch API: up to 500 companies per file, answers applied when they come back.
2–5 decision makers per company are saved with name, title and seniority.

Limit 1,500 companies a day · about $3 per 1,000 companies (jina + GPT in batch mode) · answers within a few hours

3Website scan

linkedin-email-finder, part A · through the VPS proxy

The company website (home, /contact, /about) is read through the VPS proxy, never from the office IP.
Gives the phone number and the e-mail addresses printed on the site (info@ …).
If a printed address belongs to a known person, the company's e-mail pattern is learned from it (e.g. first.last@).

Output: company phone + printed addresses · free

4E-mail for every person

linkedin-email-finder, part B · verifier on verify.reachiq.io

Find the domain's pattern: cache → production's 14M e-mails (how people on this domain write theirs) → what the website scan learned.
Pattern known → 1 guess; unknown → up to 6 guesses (first.last, first, flast …).
Each guess is checked with the company's mail server by the VPS verifier (SMTP RCPT; nothing is ever sent): valid / invalid / accept-all / unknown.
Valid → saved and the pattern cached for the next person. Accept-all → the best guess is kept with status accept-all.

Limit 8,000 checks a day · free

One company, end to end: Y-Säätiö (Finland)

A real company from the crawl, with the real index names. Three services touch it, one after another. They never write to production; they only read it.

Day 1, 10:00 — `linkedin-chain-crawler`

Takes the next pending company from the linkedin_chain_companies index and opens linkedin.com/company/ysaatio through the AWS gateway.
Fills the company document: company_name Y-Säätiö, company_website ysaatio.fi, industry Non-profit, 86 employees, Helsinki, Finland … crawl.status = done.
The page shows an "Employees at Y-Säätiö" box with 3 random people (LinkedIn picks them, no titles). The crawler creates 3 documents in linkedin_chain_persons:

full_name	job_title	crawl.source	company_id
Marika Kyrölä	—	company_page	6abce…
Tommi Kaipainen	—	company_page	6abce…
Inari Virkkala	—	company_page	6abce…

The crawler does not know who these people are — CEO or intern. It only has names and profile URLs.

Day 2, 03:00 — `linkedin-people-search`

Takes Y-Säätiö from linkedin_chain_companies (its crawl.people_search is still empty) and sends 3 searches to jina.ai, with the header X-Site: linkedin.com/in so only LinkedIn profile pages come back:

#	Query	Results
1	`"Y-Säätiö" CEO OR founder OR "co-founder" OR owner OR "managing director" OR president OR partner`	10
2	`"Y-Säätiö" perustaja OR toimitusjohtaja OR omistaja OR CEO` (Finnish titles, because country = Finland)	10
3	`"Y-Säätiö" "ysaatio.fi" CEO OR founder OR director` (with the website domain, to separate namesakes)	10

Each result has three things — exactly what a Google result shows, nothing more:
url: linkedin.com/in/ojankoski-teija-a3b9123
title: Teija Ojankoski - CEO - Y-Säätiö | LinkedIn
snippet: Toimitusjohtaja, Y-Säätiö · Helsinki, Finland · 500+ connections…
30 results, duplicates removed (same profile URL = same person; Teija came up in all three searches and is counted once) → 18 candidates. "Candidates" means: maybe Y-Säätiö people — some can be former employees, a different "Y-Foundation", or someone who only mentioned the company.
The 18 (url + title + snippet) are stored on the company document in crawl.people_search.candidates_json, status awaiting_judge, and go into the next OpenAI batch file with up to 499 other companies.
GPT-6 Luna receives the company's identity (name, ysaatio.fi, Helsinki, Finland, Non-profit) plus the 18 lines, and answers two questions per line:

Candidate line	Same company?	Decision maker?	Result
#3 Teija Ojankoski - CEO - Y-Säätiö · Helsinki	yes	yes	accept · c_level
#7 Inari Virkkala - Project Manager - Y-Säätiö	yes	no	title only
#12 … - Former CEO Y-Säätiö (2015–2019)	no (former)	—	reject
#15 … - CEO @ Y-Foundation USA	no (other company)	—	reject

5 of 18 pass both questions. The worker now looks them up in linkedin_chain_persons by contact_linkedin_url: Teija, Jouni, Sari, Saija, Juha are not there → 5 new documents with job_title, crawl.seniority, crawl.source = search. Inari Virkkala is already there (from the crawler) → no new document, her existing one gets job_title = Project Manager, crawl.title_source = search.

full_name	job_title	crawl.source	crawl.seniority
Marika Kyrölä	—	company_page	—
Tommi Kaipainen	—	company_page	—
Inari Virkkala	Project Manager (updated)	company_page	—
Teija Ojankoski	CEO	search	c_level
Jouni Isomöttönen	Kiinteistö- ja rakennuttamisjohtaja	search	vp_director
Sari Timonen	Development Director	search	vp_director
Saija Turunen	Head of Research	search	head
Juha Kahila	Head of International Affairs	search	head

On the company: crawl.people_search = {status: done, searches: 3, candidates: 18, accepted: 5, batch_id: …}. We never open a profile and never ask LinkedIn — we only read search-result titles, like noting the names on Google's first page for "Y-Säätiö CEO".

Day 2, 09:00 — `linkedin-email-finder`

Website scan: takes Y-Säätiö from linkedin_chain_companies (its crawl.website_scan is empty), reads ysaatio.fi through the VPS proxy → writes company_phone_number and crawl.website_scan.
E-mail per person: takes the 8 documents in linkedin_chain_persons whose email is empty. Pattern lookup: linkedin_chain_email_patterns has nothing for ysaatio.fi → production primary_records (read-only) has no ysaatio.fi address → 6 guesses for the first person → the VPS verifier confirms marika.kyrola@ysaatio.fi → pattern first.last is cached in linkedin_chain_email_patterns → the other 7 need one guess each.

full_name	email	email_verification_status
Marika Kyrölä	marika.kyrola@ysaatio.fi	valid
Jouni Isomöttönen	jouni.isomottonen@ysaatio.fi	valid
Sari Timonen	sari.timonen@ysaatio.fi	valid
Saija Turunen	saija.turunen@ysaatio.fi	valid
…	7 valid in total

Who writes which key: the crawler and the search create documents (names, company fields, title); the e-mail finder only updates email, email_verification_status, email_verification_updated_at and company_phone_number. No worker overwrites another worker's keys.

Where everything lives

Elasticsearch index	What	Format
`linkedin_chain_companies`	companies + the crawl queue	same fields as `primary_companies_1` + `crawl`
`linkedin_chain_persons`	people (from the crawler and from the search)	same fields as `primary_records` + `crawl`
`linkedin_chain_email_patterns`	e-mail pattern per domain (cache)	—
`linkedin_chain_batches`	OpenAI batch records	—
`primary_companies_1`, `primary_records`	ReachIQ production	read-only for every worker

The dashboard reads linkedin_chain_companies and linkedin_chain_persons every 15 seconds. The People tab's "Source" column shows company page or search for each person.

The e-mail finder in detail

One service, one loop, three jobs in turn every round. Everything on company websites and mail servers goes through the VPS, never the office IP. The whole stage is free except the website-extra part.

0Website extra — 5 companies per round

For companies whose extra.website_extracted_at is empty: home page + careers page + up to 2 about/services pages through the VPS proxy.
GPT-6 Luna fills the fixed keys: extra.services, extra.short_description, extra.careers_url, extra.hiring, extra.open_roles_on_site.

Limit 2,000 companies a day · ~$0.001 per company

1Website scan — 10 companies per round

For companies whose crawl.website_scan is empty: 3 pages (/, /contact, /about) read through the VPS proxy.
From the text and the tel: / mailto: links: the phone number → company_phone_number; the addresses printed on the site → crawl.website_scan.emails.
If a printed address belongs to a person we already know (e.g. l.gomez@ and Laurent Gomez is in the persons index) the domain's pattern is learned (fi.last) and cached in linkedin_chain_email_patterns.

Free

2E-mail per person — 20 people per round, decision makers first

people in linkedin_chain_persons whose email is empty and whose crawl.email_search has not run yet

person: Marika Kyrölä @ ysaatio.fi
  ├─ skip?  free-mail domain (gmail…) / no company website / no first name  → status skipped, nothing tried
  ├─ find the domain's pattern (domain level, stops at the first hit):
  │    1. cache  linkedin_chain_email_patterns has ysaatio.fi?        → yes: use it, stop     (1 ES get)
  │    2. production  primary_records has addresses "*@ysaatio.fi"?  → yes: learn the pattern from them, cache it, stop
  │                                                                   → no: cache "prod: nothing" so production is never asked again
  │    3. website scan learned one? (already in the cache, so it is found in step 1)
  ├─ pattern known   → 1 guess (+2 fallbacks only if the mail server rejects it)
  │  pattern unknown → 6 guesses in order: marika.kyrola · marika · mkyrola · marikakyrola · m.kyrola · marika_kyrola
  ├─ each guess goes to the VPS verifier: GET /verify?email=…   (SMTP RCPT TO, nothing is ever sent)
  │    marika.kyrola@ysaatio.fi → valid ✅ → stop
  ├─ save: email, email_verification_status = valid, email_verification_updated_at,
  │        crawl.email_search = {pattern: first.last, pattern_source, tried: […], status: valid, checked_at}
  └─ cache the pattern "first.last" for ysaatio.fi (source: verified) → the other 7 people need one guess each

Verifier answer	What the finder does
valid	stop, save the address, cache the pattern for the domain
invalid	try the next guess (never stored)
accept-all	keep this guess with status accept-all, stop (more guesses would all say OK)
unknown twice	stop; nothing stored, unless the pattern was known: then the best guess is kept with status unknown

Y-Säätiö: Marika (first person) cache ✗ → production ✗ → 6 guesses → #1 valid → pattern cached. Jouni, Sari, Saija … (the other 7): cache ✓ → 1 guess each. Metabase: production already had cynthia@, sameer@ → pattern "first" → 1 guess, valid. So production is asked once per domain and most people cost one verifier call — that is why 8,000 checks a day are easy.

Limits and safety

8,000 checks a day (EMAIL_DAILY_LIMIT, raised from 3,000 on 7 Oct); then it waits for 00:05 UTC.
The verifier's /health says whether port 25 is open; if not, only website scans run until it is.
Verifier or GPT unavailable → the worker pauses 30 minutes and retries; nothing is marked failed.
Who writes which key: the crawler and the search create documents (names, company fields, title); the e-mail finder only updates email, email_verification_status, email_verification_updated_at, company_phone_number and extra.*. No worker overwrites another worker's keys.
Cost: free, except website extra (~$2 a day at full speed). Status on 12 Oct: 31,749 people checked → 6,058 valid, 9,967 accept-all; 4,593 phones; 9,315 websites scanned.

All keys we store

From a live sample of 3,000 documents per index (12 Oct 2026). "Filled" = how often the key has a value. Every top-level key is a ReachIQ production field; our own bookkeeping sits under crawl and never goes to production.

Company — index `linkedin_chain_companies`

format: primary_companies_1

Key	Filled	From
`company_name`	100%	LinkedIn page
`company_linkedin_url`	100%	LinkedIn page
`company_linkedin_industry`	97%	LinkedIn page
`industry`	97%	copy of the above (production convention)
`organization_type`	96%	LinkedIn page
`company_linkedin_description`	90%	LinkedIn page
`company_website`	87%	LinkedIn page
`original_company_website`	87%	copy of website
`number_of_employees`	87%	LinkedIn page
`company_address`	83%	LinkedIn page (Locations)
`company_city`	82%	split from address
`company_country`	82%	split from address
`company_state`	66%	split from address
`number_of_follower`	65%	LinkedIn page
`company_founded_on`	64%	LinkedIn page
`company_specialities`	47%	LinkedIn page
`company_phone_number`	42%	website scan (e-mail finder)
`sanitize_stage` = 39	100%	fixed (production convention)
`created_at`, `updated_at`	100%	crawler
`data_source` = linkedin_chain_crawler	100%	extra: marks the origin

extra.* — fixed extra details, same keys for every company (null / [] when not found)

Key	From
`extra.open_jobs` (int)	LinkedIn page: "N open jobs" — hiring signal
`extra.products` (list of name, category, url)	LinkedIn page: Products section
`extra.services` (list, max 10)	company website, GPT-6 Luna
`extra.short_description`	company website, GPT-6 Luna (one line, English)
`extra.careers_url`	company website: careers / jobs page link
`extra.hiring` (true / false / null)	careers page lists open roles?
`extra.open_roles_on_site` (int)	roles counted on the careers page
`extra.linkedin_extracted_at`, `extra.website_extracted_at`	when each part was read

crawl.* — bookkeeping only, not sent to production

status, depth, attempts, queued_at, crawled_at, updated_at, discovered_from, final_url, similar_pages, affiliated_pages, employees_preview_count, name_hint, industry_hint, last_error, known_in_prod, prod_id, company_size (97%), company_headquarters (82%), postal_code (62%), website_scan{…} (87%), people_search{…} (87%)

Production keys we never fill (not available from these sources)

funding_amount, funding_round, funding_time, revenue, annual_revenue, company_facebook_url, company_twitter_url, company_keywords, company_technologies, seo_description, saved_tenant_ids, company_dynamic_address, old_website

Person — index `linkedin_chain_persons`

format: primary_records

Key	Filled	From
`first_name`	100%	crawler (page) or search
`last_name`	99%	crawler or search
`full_name`	100%	crawler or search
`contact_linkedin_url`	100%	crawler or search
`company_id`	100%	link to the company document
`job_title`	34%	search (decision makers)
`email`	32%	e-mail finder
`email_verification_status`	32%	e-mail finder: valid / accept-all / unknown
`email_verification_updated_at`	32%	e-mail finder
`company_name`, `company_linkedin_url`	100%	copied from the company
`number_of_employees`	99%	copied
`company_linkedin_industry`, `industry`	94%	copied
`organization_type`	91%	copied
`company_linkedin_description`	85%	copied
`company_address`, `company_city`	81%	copied
`company_country`	79%	copied
`company_state`	65%	copied
`number_of_follower`	64%	copied
`company_website`, `original_company_website`	58%	copied
`company_founded_on`	57%	copied
`company_specialities`	42%	copied
`sanitize_stage` = 39, `created_at`, `updated_at`	100%	fixed / whoever wrote last
`data_source`	100%	extra: marks the origin

crawl.* — bookkeeping only

source (company_page / search, 100%), company_crawl_url (100%), seniority (32%, search people), title_source, email_search{status, domain, pattern, pattern_source, tried, checked_at} (58%)

Production keys we never fill

direct_phone_number, mobile_number, corporate_phone_number, person_city, person_state, person_country, person_dynamic_address, company_logo, funding_*, revenue, company_keywords, company_technologies, seo_description, saved_tenant_ids

How "valid" and "accept-all" are decided

The verifier on the VPS asks the company's own mail server, without sending any mail.

The check

Look up the domain's MX record: which mail server handles it (Google, Microsoft, Zoho, the company's own).
SMTP conversation with that server: HELO → MAIL FROM → RCPT TO: name@company.com ("would you accept mail for this address?") → QUIT. Nothing is ever sent.
250 OK → mailbox exists → valid
550 user unknown → does not exist → invalid (never stored, next guess is tried)
250 for every address, even a random made-up one we test too → the server will not tell → accept-all
No answer / blocked / greylisting → unknown

What the statuses mean for campaigns

Valid: the mail server itself confirmed the mailbox. Most reliable; bounces about 1–2%.
Accept-all: not wrong, but not confirmed. Google Workspace, Microsoft 365 and Proofpoint domains (about 60% of ours) never tell an unknown sender, so we keep the best guess by pattern (first.last@ …, learned from production's 14M addresses). Bounces 10–20%.
Use valid first; accept-all with lower priority and a warmed-up sender. That is why the dashboard shows them separately.

Daily output and cost

At full speed with today's limits, all four workers running. Measured figures come from the live run between 30 September and 12 October 2026. The GPT judge runs through the OpenAI Batch API (half price, answers within a few hours).

What comes out per day

New companies	~2,000	limit 2,000 pages · measured 2,000/day
People from company pages	~4,000	name + profile, no title · ~2 per company
Decision makers	~2,700	1,500 companies/day × 1.8 found per company (measured 7–12 Oct; a third of companies give none)
E-mail checks	8,000 people	limit raised 7 Oct · measured on 31,700: 19% valid · 31% accept-all · 50% nothing
Verified e-mails	~1,500	plus ~2,500 best-guess (accept-all) addresses, when enough people are waiting
Company phone numbers	~1,000	about half of all websites show one

What it costs

Service	Per day	Per month
AWS API Gateway — IP rotation, 2,000 requests	$0.10	$3
jina.ai Search — 1,500 companies × 3 searches	$2.25	$70
OpenAI GPT-6 Luna — judge, 1,500 companies, Batch API	$0.72	$22
VPS verify.reachiq.io (t3.micro)	$0.27	$8
Website scan + e-mail verification	$0	$0
Total	~$5	~$150

≈ $2.50 per 1,000 new companies, decision makers and e-mails included

A full month (30 days)

New companies	60,000
People from company pages	120,000
Decision makers	~80,000
Verified e-mails	~25,000–35,000
Best-guess e-mails (accept-all)	~40,000–55,000
Phone numbers	~30,000

Decision-maker search runs at 1,500/day while the crawler brings 2,000/day: the backlog grows by 500 a day unless the limit is raised (+33% jina/GPT cost, about $200/month).

What needs recharging

jina.ai	~1.35 billion tokens/month ≈ $70. Buy tokens at jina.ai → API → Billing; the key stays the same. When the balance is empty the decision-maker search pauses by itself and resumes after a recharge.
OpenAI	~$70/month on the ICP project key (Batch API), plus $4–5 per 1,000 Lead Finder companies. platform.openai.com → Billing: keep a balance or auto-recharge; Limits: monthly budget $150.
AWS	Gateway + VPS ≈ $11/month on the normal AWS bill. Nothing to recharge; a $25 budget alert is enough.

LinkedIn Crawler

Crawl status

Top industries (crawled)

Crawled companies

Details

Costs so far

Balances

Last 14 days, per day

Start crawling from a LinkedIn company URL

Seeds and their chains

Campaigns

Campaigns

How it works

In plain words: what the LinkedIn crawler does

What data the crawler gives us

1Find companies

2Decision makers

3Website scan

4E-mail for every person

One company, end to end: Y-Säätiö (Finland)

Day 1, 10:00 — `linkedin-chain-crawler`

Day 2, 03:00 — `linkedin-people-search`

Day 2, 09:00 — `linkedin-email-finder`

Where everything lives

The e-mail finder in detail

0Website extra — 5 companies per round

1Website scan — 10 companies per round

2E-mail per person — 20 people per round, decision makers first

Limits and safety

All keys we store

Company — index `linkedin_chain_companies`

Person — index `linkedin_chain_persons`

How "valid" and "accept-all" are decided

The check

What the statuses mean for campaigns

Daily output and cost

What comes out per day

What it costs

A full month (30 days)

What needs recharging