spydernet
every page she reads is a strand you can verify.
Real browsers crawl the crypto web. What is new gets hashed, sealed under a merkle root, and pretrains WIDOW.
0pages kept
0novel tokens
0hosts
0kept / min
CA
0x91f254a75263eb9379b8fa065e8f5d92cde9602athe page as the spyder parses it ยท green: new to the dataset ยท grey: already read ยท red: links it will follow
| pid | spyder | s | chapter | pages | novel | rej | url |
|---|
silk log
widow ยท pipelinea stage is done only when its artefact exists
chapters
ask what she has readretrieval with citations ยท no weights yet
open epoch
Shares follow novel tokens ร relevance, not page count. Re-reading known text earns nothing.
| spyder | pages | novel tok | share |
|---|
sealed epochs
none sealed yet
verify a pageruns in your browser
Pick any page from the silk log or the web, or type its id. Your browser re-hashes the text, the receipt and the merkle path.
spawn a spyderopen spawn ยท the burn gate is not live
Name it and point it at a site. It reads that host only, obeys robots.txt and stops at the first bot check. Its pages are credited to its name on the board.
the hive
| spyder | chapter | state | pages | novel tok | rejected |
|---|
SPYDERNET(1) Hive Manual SPYDERNET(1)
NAME
spydernet โ a provable crawl of the crypto web, and the model pretrained on it.
PIPELINE
crawl โ gate โ dedupe โ novelty โ tokenize โ receipt โ seal โ export โ pretrain
DESCRIPTION
crawl Spyders are real headless Chromium. One polite visit per host every few
seconds, robots.txt obeyed, and a host is dropped at the first bot check.
The frontier prefers the hosts read least, so coverage spreads.
gate A page is kept only if it is long enough and actually about crypto.
novelty The page is cut into 5-word shingles and compared with everything already
read. Its novel tokens are the share of it that is new. Under 35% new, it is
rejected as a near-duplicate.
receipt {url, sha256(text), tokens, novel, score, spyder, ts}. The receipt hash is a
leaf. Leaves are hash-chained as they land, and merkle-rooted per epoch.
seal When an epoch closes its root is fixed. contracts/SpyderNet.sol stores one
root per epoch on-chain, write-once, and can check a
page's proof on-chain. The contract is written and compiles; not deployed.
rewards An epoch's pool is split by novel tokens ร relevance among spyders with
enough pages. Shares are computed here; no pool is wired to them yet.
widow The tokenizer is trained on the crawl. The export is plain uint16 shards.
train/train.py pretrains from random init and stores the dataset root in
the checkpoint, so weights name the data they saw.
ask Until weights exist, ask is retrieval: the passages she read, with sources
and proofs.
FILES
/v1/live websocket: spyder state, browser frames, pages, rejects, epochs
/v1/stats totals, chapters, frontier, open epoch root, chain head
/v1/web the dataset as a graph of pages and links
/v1/pages/:id one page's text, as widow will read it
/v1/proof/:id receipt, leaf, merkle path and root for one page
/v1/epochs the open board and every sealed epoch
/v1/ask POST {question} โ cited passages
/v1/spawn POST {name, seed} โ a new spyder
COMMANDS
npm start run the hive and this site
npm run tokenizer train the BPE tokenizer on the crawl
npm run export write train.bin, val.bin, manifest.json
python train/train.py pretrain widow on the export
spydernet 0.1 SPYDERNET(1)