man · lurk

NAME

lurk — pretraining a crypto-native llm from scratch, on nothing but what its lurkers read.

SYNOPSIS

lurk → ingest → map → train moar-v{n}the loop. moar-v0 trains at 2.0M tokens; each next version when the crawl doubles.

order url --burn 10,000 $LURKsend the lurkers to a site.

adopt lurker --burn 250,000 $LURKown a lurker, earn from it.

epochs --every 12hsplit the pool, in SOL.

DESCRIPTION

lurk is a crawler and a language model that knows nothing but what the crawler read. a fleet of lurkers reads the crypto web in real browsers; what they read becomes the dataset; the dataset becomes moar, named after the old advice, lurk moar. every screen the lurkers see is streamed live, and every version of moar is public.

lurk: 64 lurker identities, each a headless chromium when awake. a few are awake at a time, owned ones first; they read docs, whitepapers, research blogs, forums and wikis. they identify as lurkbot, obey robots.txt and its crawl-delay, wait at least 4 s between pages of one domain, never log in, and decline cookie banners. what they see streams to /lurkers.
ingest: each page loses its chrome: the main text is extracted (readability), normalized, and checked. it has to be english, mostly prose, about crypto, and at least 500 characters. exact duplicates (sha1) and near duplicates (64-bit simhash, hamming ≤ 3) are dropped. what survives is accepted; everything else is still logged, with the reason.
map: the dataset is indexed by project, a seed source like solana docs, and chapter, the first path segment under the project's base url. coverage is pages read over pages discovered, per project, so it can reach 100%.
train: moar is pretrained from scratch on the crawl and nothing else. first a byte-level bpe tokenizer is trained on the crawl, then a small gpt. moar-v0 trains once the crawl holds 2.0M tokens; each next version when the crawl has doubled since the last one (1.3× after a day). every version starts from zero on the bigger dataset and is bigger than the last: ~5m parameters, then ~14m, ~42m, and up as the crawl grows. the weights, the tokenizer and a model card are downloadable from /moar.
order url: anyone can burn 10,000 $LURK to send the lurkers somewhere: up to 25 new pages of that site within 3 h, with a public report. checks come first: robots.txt, a real page, crypto, no scam, adult or gambling. nothing is burned until they pass. the burn carries the memo lurk:order:<id>; an order waits 30 minutes for it. /order
adopt lurker: an unowned lurker can be adopted by burning 250,000 $LURK with the memo lurk:adopt:<id>. the burning wallet becomes its owner and earns epoch payouts from then on.
epochs: time is cut into 12 h epochs that close at 00:00 and 12:00 utc; epoch n = floor(unix_ms / 43,200,000). an owned lurker that keeps 25+ new accepted pages in an epoch, counted from its adoption, is eligible for that epoch's payout.

ECONOMICS

pump.fun creator fees for $LURK land in the queen wallet, which opens at launch. 60% pays for compute: crawling and training. 40% is the epoch pool for lurker owners.

fees_in   = Σ creator fees that landed in the epochpool      = floor(fees_in × 0.4)eligible  = owned lurkers with ≥ 25 new accepted pages in it (counted from adoption)each      = floor(pool / eligible)  # lamports, per lurker

payouts are prepared by code at the close and sent by a human with their own key, memo lurk:payout:<n>. lurk itself never sends a transaction; burns are signed by you, in your wallet. every SOL in and out is on the public ledger at /treasury.

EXAMPLES

the live numbers

$ curl -s https://api.uselurk.xyz/v1/stats

{
  "lurkers_awake": 2,
  "pages_read": 11347,
  "dataset_tokens": 21220418,
  "domains": 79,
  "sol_for_compute": 0,
  "epoch": {"n": 41470, "closes_at": 1791547200000, "eligible": 0},
  "moar": {"version": 3, "params": 5310720, "state": "waiting"},
  "launched": false,
  …
}

give moar the start of a sentence

$ curl -s -X POST https://api.uselurk.xyz/v1/moar/ask \
    -H 'content-type: application/json' \
    -d '{"question":"a validator is"}'

answers {answer, version, tokens, ms, sources}. until moar-v0 exists: 503 {error:"no_model"}, still with sources. moar is a base model: it continues text, it doesn't chat. the sources are pages it read about the same thing. 6 questions a minute, 400 characters each.

order a crawl (runs the checks)

$ curl -s -X POST https://api.uselurk.xyz/v1/orders \
    -H 'content-type: application/json' \
    -d '{"url":"https://docs.example.xyz"}'

200 → an order awaiting_burn with its memo, amount and expiry. 422 → rejected, with the failed check. 503 → orders open at launch. after the burn, POST /v1/orders/<id>/confirm {signature} queues it.

follow everything live

$ websocat wss://api.uselurk.xyz/v1/live

first a hello with the full state, then stats, lurker, page, log, train, ledger, order and epoch as they happen. send {"t":"sub","frames":[3]} for a lurker's screens. a ping every 20 s; answer {"t":"pong"}.

FILES

everything is public, json, under https://api.uselurk.xyz. errors look like {error, detail?}.

GET/v1/health: ok, version, uptime, crawler state
GET/v1/config: token mint, queen wallet, burn amounts, launched
GET/v1/stats: the live numbers
GET/v1/lurkers: every lurker slot, awake or asleep
GET/v1/lurkers/:id: one lurker + its last 30 pages
GET/v1/pages?limit=&before=: pages read, newest first, with status
GET/v1/pages/:id: one page + the first 1,200 chars of its text
GET/v1/map: projects, chapters, coverage
GET/v1/web?limit=: the domain link graph
GET/v1/search?q=: full-text search over accepted pages
GET/v1/moar: training state + every version
POST/v1/moar/ask: {question} → moar continues it (6/min)
GET/v1/moar/metrics?version=: loss curve
GET/v1/moar/weights/:v/:file: model.safetensors, tokenizer.json, config.json, card.md
GET/v1/treasury: queen wallet balance, fees, pool, payouts
GET/v1/treasury/ledger: every SOL in and out
GET/v1/treasury/epochs: epochs and their payouts
POST/v1/orders: {url} → checks → an order awaiting its burn
POST/v1/orders/:id/confirm: {signature} → verifies the burn, queues it
GET/v1/orders: recent orders
GET/v1/orders/:id: the public report
POST/v1/lurkers/:id/adopt: {signature} → verifies the adopt burn
GET/v1/frames/:id.jpg: a lurker's latest screen
WS/v1/live: everything, as it happens (see EXAMPLES)

BUGS

moar is small, and it shows: a base model trained on one crawl will say confident nonsense. lurkers skip what they can't read: pages behind logins, paywalls, captchas, and apps that are all javascript and no text.

lurk(1)