← 返回阅读笔记 / Back to reading note
来源:作者原文。2026-09-07抓取,原始行列,不是统一复跑成绩。
| Method | CR (# LU) | Accuracy |
|---|---|---|
| BM25 Retrieval | ||
| Top-1 | 89.27% (1) | 70.34% 0.06 |
| Top-2 | 78.96% (2) | 79.05% 0.05 |
| Top-3 | 68.50% (3) | 82.65% 0.05 |
| Top-4 | 58.57% (4) | 84.42% 0.13 |
| Neural Retrieval with Gemini API | ||
| Top-1 | 89.91% (1) | 71.32% 0.19 |
| Top-2 | 80.08% (2) | 79.02% 0.10 |
| Top-3 | 70.28% (3) | 83.41% 0.10 |
| Top-4 | 60.68% (4) | 84.88% 0.03 |
| Full Raw Content | 0% | 85.83% 0.19 |
| GistMem | 85.53% | 77.52% 0.13 |
| ReadAgent-P | ||
| Look up 1 pg | 76.00% (1.0) | 84.13% 0.10 |
| Look up 1-2 pgs | 72.17% (1.6) | 86.16% 0.12 |
| Look up 1-3 pgs | 69.36% (2.0) | 86.59% 0.10 |
| Look up 1-4 pgs | 67.73% (2.2) | 86.86% 0.00 |
| Look up 1-5 pgs | 66.45% (2.3) | 86.83% 0.10 |
| Look up 1-6 pgs | 64.75% (2.5) | 86.91% 0.08 |
| ReadAgent-S 1-6 pgs | 58.53% (3.2) | 87.17% 0.18 |
| Gutenberg Validation (58 docs & 1743 questions) | Gutenberg Test (177 docs & 5207 questions) | |||||||||||
| Method | CR (# LU) | LR-1 | LR-2 | R-1 | R-2 | R-L | CR (# LU) | LR-1 | LR-2 | R-1 | R-2 | R-L |
| BM25 Retrieval | ||||||||||||
| Top-1 | 97.63% (1) | 39.01% | 50.14% | 0.166 | 0.061 | 0.156 | 97.42% (1) | 43.5% | 55.33% | 0.176 | 0.065 | 0.165 |
| Top-2 | 95.24% (2) | 49.34% | 60.76% | 0.203 | 0.079 | 0.191 | 94.80% (2) | 51.70% | 64.53% | 0.206 | 0.082 | 0.194 |
| Top-3 | 93.34% (3) | 52.73% | 63.68% | 0.208 | 0.080 | 0.195 | 93.02% (3) | 52.97% | 66.03% | 0.210 | 0.083 | 0.197 |
| Top-4 | 92.47% (4) | 53.59% | 64.26% | 0.211 | 0.082 | 0.197 | 92.27% (4) | 53.60% | 66.16% | 0.210 | 0.084 | 0.197 |
| Neural Retrieval with Gemini API | ||||||||||||
| Top-1 | 98.19% (1) | 34.25% | 46.53% | 0.146 | 0.051 | 0.134 | 98.14% (1) | 36.47% | 47.8% | 0.150 | 0.054 | 0.140 |
| Top-2 | 96.30% (2) | 44.69% | 54.96% | 0.180 | 0.069 | 0.167 | 96.15% (2) | 44.48% | 56.17% | 0.182 | 0.070 | 0.170 |
| Top-3 | 94.62% (3) | 46.24% | 57.31% | 0.191 | 0.077 | 0.178 | 94.42% (3) | 48.97% | 60.73% | 0.195 | 0.076 | 0.183 |
| Top-4 | 93.45% (4) | 48.59% | 59.21% | 0.196 | 0.079 | 0.184 | 93.25% (4) | 50.62% | 62.05% | 0.203 | 0.080 | 0.191 |
| GistMem | 96.89% | 55.31% | 68.22% | 0.233 | 0.091 | 0.218 | 96.80% | 55.79% | 71.19% | 0.231 | 0.092 | 0.217 |
| ReadAgent-P | ||||||||||||
| Look up 1 pg | 95.15% (0.94) | 58.92% | 71.89% | 0.244 | 0.101 | 0.230 | 94.84% (0.93) | 59.98% | 73.23% | 0.240 | 0.098 | 0.226 |
| Look up 1-2 pgs | 94.79% (1.23) | 59.84% | 72.29% | 0.239 | 0.098 | 0.224 | 94.36% (1.34) | 59.19% | 72.65% | 0.231 | 0.091 | 0.218 |
| Look up 1-3 pgs | 94.39% (1.50) | 59.84% | 71.89% | 0.240 | 0.098 | 0.226 | 94.03% (1.61) | 59.63% | 72.84% | 0.230 | 0.093 | 0.217 |
| ReadAgent-S 1-2 pgs | 94.35% (1.38) | 57.89% | 71.14% | 0.239 | 0.097 | 0.225 | 93.86% (1.46) | 60.48% | 72.48% | 0.232 | 0.095 | 0.219 |
| ReadAgent-S 1-3 pgs | 94.08% (1.57) | 58.52% | 71.49% | 0.242 | 0.098 | 0.229 | 93.67% (1.57) | 60.55% | 72.79% | 0.231 | 0.095 | 0.219 |
| Movie Validation (57 docs & 1699 questions) | Movie Test (172 docs & 5139 questions) | |||||||||||
| BM25 Retrieval | ||||||||||||
| Top-1 | 97.07% (1) | 32.67% | 42.61% | 0.156 | 0.058 | 0.144 | 96.61% (1) | 33.64% | 43.34% | 0.154 | 0.054 | 0.143 |
| Top-2 | 94.12% (2) | 39.97% | 50.21% | 0.187 | 0.070 | 0.174 | 93.81% (2) | 42.50% | 53.05% | 0.191 | 0.072 | 0.178 |
| Top-3 | 91.18% (3) | 43.61% | 53.91% | 0.198 | 0.077 | 0.185 | 91.00% (3) | 46.97% | 57.52% | 0.207 | 0.080 | 0.193 |
| Top-4 | 88.24% (4) | 46.85% | 57.62% | 0.210 | 0.084 | 0.198 | 88.19% (4) | 50.18% | 60.13% | 0.217 | 0.085 | 0.202 |
| Neural Retrieval with Gemini API | ||||||||||||
| Top-1 | 97.07% (1) | 32.02% | 41.44% | 0.153 | 0.053 | 0.142 | 96.67% (1) | 37.24% | 46.22% | 0.130 | 0.043 | 0.118 |
| Top-2 | 94.19% (2) | 43.20% | 51.38% | 0.160 | 0.057 | 0.148 | 93.90% (2) | 46.49% | 54.60% | 0.164 | 0.061 | 0.151 |
| Top-3 | 91.29% (3) | 47.56% | 56.21% | 0.176 | 0.064 | 0.163 | 91.14% (3) | 50.69% | 58.92% | 0.186 | 0.071 | 0.172 |
| Top-4 | 88.38% (4) | 49.09% | 59.33% | 0.193 | 0.075 | 0.180 | 88.36% (4) | 52.13% | 59.41% | 0.184 | 0.072 | 0.171 |
| GistMem | 92.09% | 52.56% | 64.39% | 0.242 | 0.103 | 0.227 | 91.98% | 54.68% | 64.00% | 0.248 | 0.105 | 0.234 |
| ReadAgent-P | ||||||||||||
| Look up 1 pg | 89.20% (0.99) | 53.38% | 65.57% | 0.247 | 0.106 | 0.233 | 89.22% (0.98) | 57.68% | 68.01% | 0.274 | 0.116 | 0.260 |
| Look up 1-2 pgs | 87.68% (1.52) | 54.62% | 65.63% | 0.238 | 0.098 | 0.223 | 88.10% (1.39) | 58.24% | 68.81% | 0.270 | 0.115 | 0.255 |
| Look up 1-3 pgs | 86.57% (1.91) | 54.91% | 65.86% | 0.241 | 0.099 | 0.225 | 86.73% (1.89) | 58.82% | 69.12% | 0.272 | 0.116 | 0.257 |
| ReadAgent-S 1-2 pgs | 86.36% (1.98) | 59.33% | 68.28% | 0.203 | 0.082 | 0.188 | 85.92% (1.98) | 63.33% | 72.06% | 0.214 | 0.086 | 0.199 |
| ReadAgent-S 1-3 pgs | 83.56% (2.95) | 59.45% | 68.81% | 0.210 | 0.087 | 0.195 | 83.18% (2.95) | 64.53% | 73.06% | 0.217 | 0.090 | 0.202 |
| Method | CR (# LU) | LLM Rating-1 | LLM Rating-2 | ROUGE-1 | ROUGE-2 | ROUGE-L | Resp. Length |
|---|---|---|---|---|---|---|---|
| BM25 Retrieval | |||||||
| Top-1 | 95.69% (1.00) | 32.48% 1.65 | 63.85% 1.51 | 27.53 0.23 | 7.00 0.14 | 18.45 0.16 | 48.62 0.28 |
| Top-2 | 91.48% (2.00) | 29.41% 0.60 | 71.57% 1.48 | 28.85 0.17 | 7.59 0.08 | 19.34 0.14 | 52.39 0.49 |
| Top-3 | 86.93% (3.00) | 34.80% 1.14 | 79.53% 0.35 | 30.69 0.17 | 8.40 0.11 | 20.64 0.13 | 53.59 0.35 |
| Top-4 | 82.55% (4.00) | 35.66% 0.30 | 81.13% 0.35 | 31.10 0.10 | 8.53 0.06 | 20.36 0.11 | 54.96 0.42 |
| Top-5 | 78.13% (5.00) | 39.09% 0.92 | 84.44% 0.46 | 31.16 0.14 | 8.52 0.08 | 20.69 0.03 | 54.52 0.13 |
| Top-6 | 73.97% (6.00) | 37.87% 0.90 | 83.70% 0.87 | 31.06 0.04 | 8.38 0.06 | 20.43 0.08 | 56.18 0.44 |
| Neural Retrieval with Gemini API | |||||||
| Top-1 | 95.99% (1.00) | 34.80% 1.39 | 68.87% 0.62 | 27.86 0.12 | 7.12 0.04 | 18.76 0.09 | 49.46 0.23 |
| Top-2 | 92.02% (2.00) | 40.32% 0.92 | 81.50% 0.46 | 30.17 0.08 | 8.03 0.03 | 19.80 0.08 | 55.48 0.27 |
| Top-3 | 87.93% (3.00) | 40.93% 1.35 | 85.17% 1.25 | 31.36 0.12 | 8.67 0.10 | 20.68 0.10 | 56.71 0.27 |
| Top-4 | 83.71% (4.00) | 40.56% 0.62 | 84.31% 0.87 | 31.52 0.11 | 8.59 0.10 | 20.40 0.10 | 56.47 0.71 |
| Top-5 | 79.47% (5.00) | 40.20% 0.76 | 86.76% 0.60 | 31.32 0.11 | 8.49 0.11 | 20.49 0.07 | 56.73 0.91 |
| Top-6 | 75.44% (6.00) | 40.81% 0.52 | 87.01% 0.35 | 31.92 0.02 | 8.73 0.09 | 20.82 0.05 | 58.39 0.31 |
| Truncated Raw Content | |||||||
| First 6k words | 32.59% (0.00) | 14.71% 0.79 | 52.45% 0.69 | 25.42 0.05 | 4.98 0.09 | 16.58 0.10 | 58.42 0.11 |
| Last 6k words | 32.38% (0.00) | 10.42% 0.62 | 35.66% 2.46 | 20.69 0.19 | 3.44 0.10 | 14.13 0.08 | 44.23 0.11 |
| GistMem | 83.13% (0.00) | 40.20% 0.96 | 89.83% 0.76 | 31.00 0.09 | 7.99 0.04 | 20.15 0.08 | 65.75 0.20 |
| ReadAgent-P | |||||||
| Look up 1 pg | 80.00% (0.98) | 40.56% 0.46 | 89.46% 1.48 | 31.26 0.09 | 8.22 0.15 | 20.29 0.07 | 63.78 1.13 |
| Look up 1-2 pgs | 77.38% (1.71) | 39.71% 1.87 | 89.71% 0.60 | 31.11 0.04 | 8.01 0.15 | 20.21 0.04 | 64.73 1.02 |
| Look up 1-3 pgs | 75.07% (2.53) | 38.36% 1.21 | 89.71% 0.60 | 31.50 0.29 | 8.15 0.15 | 20.45 0.24 | 63.91 1.58 |
| Look up 1-4 pgs | 73.48% (3.08) | 39.95% 1.51 | 90.56% 0.35 | 31.34 0.05 | 8.08 0.18 | 20.26 0.07 | 63.40 0.79 |
| Look up 1-5 pgs | 72.29% (3.50) | 37.99% 0.96 | 87.75% 0.46 | 31.16 0.10 | 8.06 0.05 | 20.35 0.12 | 65.22 1.40 |
| Look up 1-6 pgs | 70.90% (3.97) | 39.09% 2.04 | 88.24% 0.60 | 31.50 0.30 | 8.05 0.13 | 20.26 0.13 | 66.70 0.62 |
| ReadAgent-S 1-6 pgs | 70.34% (3.55) | 46.57% 0.87 | 91.54% 0.30 | 32.90 0.17 | 8.87 0.23 | 21.15 0.14 | 68.87 0.60 |
| Method | Accuracy |
|---|---|
| GistMem + Neural Retrieval Top-1 | 82.65% |
| ReadAgent-P (Look up 1 pg) | 84.13% |
| LLM | Uniform Length | |
| ReadAgent-P (1-5 pgs) Acc. | 86.83% | 85.71% |
| GistMem | ReadAgent-P (1-5 pgs) | |||
|---|---|---|---|---|
| max_words | CR | Acc | CR | Acc |
| 400 | 81.81% | 78.91% | 66.71% | 86.82% |
| 600 | 85.53% | 77.52% | 66.45% | 86.83% |
| 800 | 88.12% | 76.22% | 65.06% | 86.34% |
| 1200 | 91.38% | 73.97% | 61.77% | 85.67% |
| Method | CR (# LU) | Accuracy |
|---|---|---|
| Neural Retrieval with Gemini API Top-3 | 73.13% (3) | 69.22% |
| Full Raw Content | 0% | 73.30% |
| GistMem | 84.24% | 66.06% |
| ReadAgent-P Look up 1-5 pgs | 76.60% (1.0) | 69.65% |
| ReadAgent-S Look up 1-6 pgs | 60.43% (3.4) | 72.10% |
| Dataset | |
|---|---|
| QuALITY | |
| QMSum | |
| NarrativeQA Gutenberg | |
| NarrativeQA movie scripts |
| Page # | Starting sentence in text |
|---|---|
| 0 | Shure and begorra… |
| 1 | The alien stooped down… |
| 2 | Interest in the horse was ended… |
| 3 | “Sure, and it’s quite a reception”… |
| 4 | Excitement, anticipation… |
| 5 | “Here he comes,”… |
| 6 | He straightened and started off… |
| 7 | The others drew back… |
| Cross-Task | Cross-Website | Cross-Domain | |||||||||||||
| CR | Ele. Acc | Op. F1 | Step SR | SR | CR | Ele. Acc | Op. F1 | Step SR | SR | CR | Ele. Acc | Op. F1 | Step SR | SR | |
| Using supervisedly trained RankLM | |||||||||||||||
| MindAct (GPT-3.5 + Rank LM∗) | – | 20.3 | 56.6 | 17.4 | 0.8 | – | 19.3 | 48.8 | 16.2 | 0.6 | – | 21.6 | 52.8 | 18.6 | 1.0 |
| MindAct (GPT-4 + Rank LM∗) | – | 41.6 | 60.6 | 36.2 | 2.0 | – | 35.8 | 51.1 | 30.1 | 2.0 | – | 37.1 | 46.5 | 26.4 | 2.0 |
| MindAct (PaLM 2-L + Rank LM∗) | – | 29.8 | 61.9 | 24.4 | 1.2 | – | 28.8 | 59.6 | 21.6 | 0.6 | – | 29.9 | 60.4 | 24.5 | 1.3 |
| No training | |||||||||||||||
| (PaLM 2-L) | |||||||||||||||
| +Raw HTML | 0.0 | 22.1 | 76.7 | 19.2 | 1.2 | 0.0 | 22.2 | 72.3 | 18.2 | 1.7 | 0.0 | 23.6 | 75.6 | 20.9 | 1.0 |
| +BM25 Retrieval (Top-1) | 43.7 | 16.3 | 61.7 | 14.2 | 0.4 | 49.7 | 17.8 | 60.8 | 15.2 | 0.0 | 51.6 | 17.3 | 60.4 | 15.9 | 0.0 |
| +BM25 Retrieval (Top-5) | 19.5 | 25.9 | 70.4 | 22.4 | 2.0 | 17.6 | 29.5 | 71.8 | 23.1 | 1.7 | 19.2 | 27.6 | 71.1 | 24.4 | 1.0 |
| +Neural Retrieval (Top-1) | 74.4 | 14.6 | 55.5 | 11.7 | 0.4 | 87.9 | 18.0 | 55.8 | 14.0 | 0.0 | 82.8 | 16.4 | 60.3 | 14.2 | 0.1 |
| +Neural Retrieval (Top-5) | 32.4 | 26.4 | 71.9 | 22.6 | 0.8 | 37.2 | 26.7 | 69.1 | 22.3 | 2.8 | 38.1 | 30.0 | 72.5 | 26.9 | 1.2 |
| GistMem | 84.4 | 11.7 | 43.1 | 9.5 | 0.0 | 82.5 | 11.7 | 43.6 | 8.4 | 0.0 | 83.0 | 13.4 | 49.6 | 11.7 | 0.5 |
| ReadAgent-P: Lookup 1 snippet | 55.1 | 31.1 | 70.1 | 26.8 | 2.0 | 54.1 | 34.5 | 74.1 | 28.2 | 2.3 | 55.2 | 36.1 | 75.6 | 33.0 | 2.0 |
| ReadAgent-P: Lookup 1-5 snippets | 35.9 | 33.7 | 72.5 | 29.2 | 2.8 | 35.6 | 37.4 | 75.1 | 31.1 | 3.4 | 48.2 | 37.2 | 76.3 | 33.4 | 2.3 |
| – | +11.6 | -4.2 | +10.0 | +1.6 | – | +15.2 | +2.8 | +12.9 | +1.7 | – | +13.6 | +0.7 | +12.5 | +1.3 | |
| – | +3.9 | +10.6 | +4.8 | +1.6 | – | +8.6 | +15.5 | +9.5 | +2.8 | – | +7.3 | +15.9 | +8.9 | +1.0 | |
| Threshold | Raw | Gist |
|---|---|---|
| 4096 Tokens | 14.2% | 88.6% |
| 8192 Tokens | 51.5% | 97.4% |
| 16384 Tokens | 79.1% | 100% |
| 50th Percentile Tokens | 8018 | 989 |
| 90th Percentile Tokens | 25337 | 3596 |
| 95th Percentile Tokens | 35779 | 5741 |
| 99th Percentile Tokens | 55642 | 12569 |
| Method | CR (# LU) | LLM Rating-1 | LLM Rating-2 | ROUGE-1 | ROUGE-2 | ROUGE-L | Resp. Length |
|---|---|---|---|---|---|---|---|
| BM25 Retrieval | |||||||
| Top-1 | 95.61% (1.00) | 24.67% 0.44 | 66.90% 0.87 | 28.81 0.13 | 8.14 0.15 | 19.62 0.18 | 48.15 0.18 |
| Top-2 | 91.32% (2.00) | 31.79% 1.31 | 79.95% 0.67 | 30.89 0.13 | 9.14 0.05 | 20.67 0.09 | 53.91 0.64 |
| Top-3 | 87.25% (3.00) | 33.45% 0.00 | 83.63% 1.05 | 31.39 0.23 | 9.11 0.05 | 21.03 0.03 | 55.15 0.61 |
| Top-4 | 82.86% (4.00) | 37.72% 1.05 | 86.12% 0.50 | 31.71 0.09 | 9.35 0.13 | 21.26 0.12 | 58.21 0.37 |
| Top-5 | 78.79% (5.00) | 39.38% 1.02 | 86.60% 0.44 | 32.66 0.04 | 9.98 0.10 | 21.86 0.05 | 59.20 1.05 |
| Top-6 | 74.62% (6.00) | 40.45% 0.89 | 90.98% 0.34 | 32.56 0.03 | 9.78 0.03 | 21.64 0.09 | 60.40 1.28 |
| Neural Retrieval with Gemini API | |||||||
| Top-1 | 95.80% (1.00) | 27.05% 0.50 | 67.97% 1.74 | 28.71 0.12 | 7.98 0.04 | 19.59 0.04 | 49.76 0.78 |
| Top-2 | 91.62% (2.00) | 35.35% 0.44 | 80.07% 0.00 | 31.65 0.18 | 9.59 0.11 | 21.29 0.11 | 56.19 0.76 |
| Top-3 | 87.39% (3.00) | 35.71% 1.37 | 88.49% 0.34 | 32.33 0.17 | 9.84 0.07 | 21.54 0.13 | 59.19 0.96 |
| Top-4 | 83.28% (4.00) | 39.62% 0.17 | 90.15% 0.34 | 32.31 0.21 | 9.69 0.15 | 21.65 0.15 | 59.86 0.11 |
| Top-5 | 79.33% (5.00) | 44.01% 0.84 | 91.22% 0.34 | 32.33 0.24 | 9.84 0.21 | 21.67 0.19 | 61.53 0.35 |
| Top-6 | 75.35% (6.00) | 44.60% 0.89 | 92.65% 0.17 | 32.55 0.08 | 9.75 0.21 | 21.39 0.13 | 61.29 0.46 |
| Truncated Raw Content | |||||||
| First 6k words | 31.51% (0.00) | 13.17% 1.05 | 47.81% 5.90 | 24.15 1.42 | 4.89 0.57 | 16.27 0.96 | 61.43 3.53 |
| Last 6k words | 33.80% (0.00) | 13.76% 0.84 | 43.42% 0.00 | 22.90 0.10 | 4.35 0.04 | 15.69 0.03 | 52.47 0.39 |
| GistMem | 82.81% (0.00) | 44.96% 0.44 | 91.93% 0.73 | 31.20 0.17 | 9.02 0.09 | 20.60 0.14 | 65.84 0.87 |
| ReadAgent-P | |||||||
| Look up 1 pg | 79.37% (0.98) | 44.84% 0.00 | 92.29% 0.34 | 31.46 0.12 | 9.09 0.11 | 20.63 0.05 | 66.74 0.74 |
| Look up 1-2 pgs | 77.00% (1.72) | 43.42% 1.01 | 92.88% 1.05 | 31.77 0.16 | 9.11 0.12 | 20.70 0.08 | 65.55 0.28 |
| Look up 1-3 pgs | 74.85% (2.46) | 44.37% 1.21 | 91.22% 0.44 | 31.89 0.06 | 8.98 0.13 | 20.70 0.09 | 66.06 1.63 |
| Look up 1-4 pgs | 73.26% (3.02) | 44.13% 0.50 | 90.51% 0.44 | 31.87 0.07 | 9.12 0.06 | 20.77 0.01 | 66.44 0.74 |
| Look up 1-5 pgs | 72.01% (3.44) | 43.42% 1.45 | 91.22% 0.60 | 31.80 0.16 | 9.03 0.07 | 20.64 0.03 | 66.48 0.39 |
| Look up 1-6 pgs | 70.65% (3.89) | 42.70% 1.54 | 90.51% 0.73 | 31.74 0.09 | 8.90 0.09 | 20.66 0.16 | 66.24 1.14 |
| ReadAgent-S 1-6 pgs | 70.75% (3.42) | 49.58% 0.44 | 93.83% 0.34 | 32.88 0.15 | 9.98 0.06 | 21.50 0.04 | 67.86 0.11 |