spydernet
every page she reads is a strand you can verify.
Real browsers crawl the crypto web. What is new gets hashed, sealed under a merkle root, and pretrains WIDOW.
0x91f254a75263eb9379b8fa065e8f5d92cde9602a| pid | spyder | s | chapter | pages | novel | rej | url |
|---|
The questions come from pages the training set never sees. The audit re-applies the hold-out rule to every page, rebuilds the training root, and checks no question leaks.
| site | pages | novel tok | new |
|---|
Shares follow novel tokens × relevance, not page count. Re-reading known text earns nothing.
| spyder | pages | novel tok | share |
|---|
Pick any page from the silk log or the web, or type its id. Your browser re-hashes the text, the receipt and the merkle path.
Name it and point it at a site. It reads that host only, obeys robots.txt and stops at the first bot check. Its pages are credited to its name on the board.
| spyder | chapter | state | pages | novel tok | rejected |
|---|
▮ has a number or an amount · ▮ names a standard, ticker or address · ▮ crypto term · ▮ hype word · ▮ the dataset has read this before
Nothing read yet.
What is built, and what comes after, in order. No dates: a phase is marked shipped only when it runs.
the web
- real-browser hive, polite by construction
- novelty gate: only what is new is kept
- receipts, merkle proofs, checked in your browser
- tokenizer trained on the crawl, dataset export
- scanner view, wall of screens, atlas
- first epoch sealed
the anchor
- hive online 24/7: the site goes live, no replay
- anchor contract deployed: every sealed root on-chain
- hold 200,000: vote on what the spyders read
widow
- first pretraining run, from random weights
- weights public, dataset root inside the checkpoint
- ask answered by widow, still with sources
- hold 200,000: api access to ask, the atlas and widow
the open hive
- stake 200,000: run a spyder on your own machine
- false receipts lose the stake
- the network checks each other's receipts
- a bigger widow every time the dataset doubles
the token · 200,000 is the key
One threshold, three uses. The balance check works today: connect a wallet below. Voting and api access are built and switch on when the hive is online. Node staking is a contract, not deployed yet.
steer
Vote on what the spyders read next: which sites, which chapters. One signed message from your wallet, no gas.
holders decide what goes into the model
access
A request quota on the api: ask, the atlas, and widow once she has weights.
checked by wallet signature, nothing to spend
run a node
Run a spyder on your own machine. Your stake is the guarantee that your receipts are real.
false receipts lose the stake
SPYDERNET(1) Hive Manual SPYDERNET(1)
NAME
spydernet — a provable crawl of the crypto web, and the model pretrained on it.
PIPELINE
crawl → gate → dedupe → novelty → tokenize → receipt → seal → export → pretrain
DESCRIPTION
crawl Spyders are real headless Chromium. One polite visit per host every few
seconds, robots.txt obeyed, and a host is dropped at the first bot check.
The frontier prefers the hosts read least, so coverage spreads.
gate A page is kept only if it is long enough and actually about crypto.
novelty The page is cut into 5-word shingles and compared with everything already
read. Its novel tokens are the share of it that is new. Under 35% new, it is
rejected as a near-duplicate.
receipt {url, sha256(text), tokens, novel, score, spyder, ts}. The receipt hash is a
leaf. Leaves are hash-chained as they land, and merkle-rooted per epoch.
seal When an epoch closes its root is fixed. contracts/SpyderNet.sol stores one
root per epoch on-chain, write-once, and can check a
page's proof on-chain. The contract is written and compiles; not deployed.
rewards An epoch's pool is split by novel tokens × relevance among spyders with
enough pages. Shares are computed here; no pool is wired to them yet.
widow The tokenizer is trained on the crawl. The export is plain uint16 shards.
train/train.py pretrains from random init and stores the dataset root in
the checkpoint, so weights name the data they saw.
ask Until weights exist, ask is retrieval: the passages she read, with sources
and proofs.
FILES
/v1/live websocket: spyder state, browser frames, pages, rejects, epochs
/v1/stats totals, chapters, frontier, open epoch root, chain head
/v1/web the dataset as a graph of pages and links
/v1/pages/:id one page's text, as widow will read it
/v1/proof/:id receipt, leaf, merkle path and root for one page
/v1/leaves/:epoch every leaf of an epoch, to rebuild its root yourself
/v1/epochs the open board and every sealed epoch
/v1/atlas what the dataset contains: standards, tickers, terms, yield per site
/v1/ask POST {question} → cited passages
/v1/read POST {text} → the text read like a page: per sentence, what is checkable, hype, already known
/v1/spawn POST {name, seed} → a new spyder
COMMANDS
npm start run the hive and this site
npm run tokenizer train the BPE tokenizer on the crawl
npm run export write train.bin, val.bin, manifest.json
python train/train.py pretrain widow on the export
spydernet 0.1 SPYDERNET(1)