Initial import: XAUUSD news sentiment models, evals, and data tooling
TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py) with its Google News crawler and DeepSeek labeller (data_collect/), the golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27 results and progress notes. Transformer weights (model.pt, model.onnx) and the crawl database are not tracked; rebuild with transformer/train.py and transformer/export_onnx.py. The DeepSeek label cache in data/ is tracked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Harwinder Sekhon
@@ -0,0 +1,6 @@
|
||||
400 weekly windows x 15 queries; 0 done, 6000 to go
|
||||
100 requests, window 2019-09-16, 3531 unique headlines, 26 req/min
|
||||
200 requests, window 2025-02-17, 7506 unique headlines, 26 req/min
|
||||
300 requests, window 2019-06-10, 11313 unique headlines, 26 req/min
|
||||
400 requests, window 2021-06-28, 15496 unique headlines, 25 req/min
|
||||
500 requests, window 2025-09-15, 19338 unique headlines, 25 req/min
|
||||
@@ -0,0 +1,20 @@
|
||||
growth: 94 windows of 30 days x 20 queries; 4245 done, 1880 to go
|
||||
100 requests, window 2023-09-13, 38522 unique headlines, 26 req/min
|
||||
200 requests, window 2020-02-01, 38742 unique headlines, 26 req/min
|
||||
300 requests, window 2019-02-06, 39126 unique headlines, 26 req/min
|
||||
400 requests, window 2022-08-19, 39452 unique headlines, 26 req/min
|
||||
500 requests, window 2025-09-02, 39819 unique headlines, 26 req/min
|
||||
600 requests, window 2023-11-12, 40298 unique headlines, 26 req/min
|
||||
700 requests, window 2026-03-31, 40734 unique headlines, 26 req/min
|
||||
800 requests, window 2025-07-04, 41074 unique headlines, 26 req/min
|
||||
900 requests, window 2020-09-28, 41336 unique headlines, 26 req/min
|
||||
1000 requests, window 2024-10-07, 41810 unique headlines, 26 req/min
|
||||
1100 requests, window 2026-06-29, 42163 unique headlines, 26 req/min
|
||||
1200 requests, window 2020-05-01, 42407 unique headlines, 26 req/min
|
||||
1300 requests, window 2021-02-25, 42948 unique headlines, 26 req/min
|
||||
1400 requests, window 2023-12-12, 43263 unique headlines, 26 req/min
|
||||
1500 requests, window 2020-06-30, 43490 unique headlines, 26 req/min
|
||||
1600 requests, window 2021-12-22, 43819 unique headlines, 26 req/min
|
||||
1700 requests, window 2022-09-18, 44095 unique headlines, 26 req/min
|
||||
1800 requests, window 2024-05-10, 44388 unique headlines, 26 req/min
|
||||
done: 1880 requests this run, 44529 unique headlines in /home/harwinder/Development/ML-NEWS/data/history.db
|
||||
@@ -0,0 +1,41 @@
|
||||
precise: 200 windows of 14 days x 21 queries; 568 done, 4200 to go
|
||||
100 requests, window 2024-07-15, 22643 unique headlines, 27 req/min
|
||||
precise: 200 windows of 14 days x 21 queries; 729 done, 4039 to go
|
||||
100 requests, window 2023-01-30, 23483 unique headlines, 27 req/min
|
||||
200 requests, window 2026-08-24, 24041 unique headlines, 26 req/min
|
||||
300 requests, window 2024-06-17, 24439 unique headlines, 26 req/min
|
||||
400 requests, window 2020-01-06, 25024 unique headlines, 26 req/min
|
||||
500 requests, window 2020-07-06, 25283 unique headlines, 26 req/min
|
||||
600 requests, window 2023-03-13, 25737 unique headlines, 26 req/min
|
||||
700 requests, window 2021-04-12, 26016 unique headlines, 26 req/min
|
||||
800 requests, window 2022-10-24, 26529 unique headlines, 26 req/min
|
||||
900 requests, window 2021-07-05, 27145 unique headlines, 26 req/min
|
||||
1000 requests, window 2026-01-26, 27546 unique headlines, 26 req/min
|
||||
1100 requests, window 2021-09-27, 28253 unique headlines, 26 req/min
|
||||
1200 requests, window 2019-06-24, 28620 unique headlines, 26 req/min
|
||||
1300 requests, window 2020-10-26, 28891 unique headlines, 26 req/min
|
||||
1400 requests, window 2024-01-29, 29175 unique headlines, 26 req/min
|
||||
1500 requests, window 2019-07-22, 29686 unique headlines, 26 req/min
|
||||
1600 requests, window 2020-09-28, 30214 unique headlines, 26 req/min
|
||||
1700 requests, window 2026-07-27, 30736 unique headlines, 26 req/min
|
||||
1800 requests, window 2024-03-11, 31133 unique headlines, 26 req/min
|
||||
1900 requests, window 2025-06-30, 31755 unique headlines, 26 req/min
|
||||
2000 requests, window 2022-06-06, 32098 unique headlines, 26 req/min
|
||||
2100 requests, window 2022-09-12, 32741 unique headlines, 26 req/min
|
||||
2200 requests, window 2020-04-13, 33019 unique headlines, 26 req/min
|
||||
2300 requests, window 2023-01-16, 33296 unique headlines, 26 req/min
|
||||
2400 requests, window 2020-03-02, 33814 unique headlines, 26 req/min
|
||||
2500 requests, window 2021-12-06, 34301 unique headlines, 26 req/min
|
||||
2600 requests, window 2024-10-21, 34714 unique headlines, 26 req/min
|
||||
2700 requests, window 2023-10-23, 35117 unique headlines, 26 req/min
|
||||
2800 requests, window 2024-12-16, 35633 unique headlines, 26 req/min
|
||||
2900 requests, window 2021-03-01, 35912 unique headlines, 26 req/min
|
||||
3000 requests, window 2021-05-24, 36290 unique headlines, 26 req/min
|
||||
3100 requests, window 2019-11-25, 36572 unique headlines, 26 req/min
|
||||
3200 requests, window 2024-02-12, 37049 unique headlines, 26 req/min
|
||||
3300 requests, window 2021-01-04, 37516 unique headlines, 26 req/min
|
||||
3400 requests, window 2022-06-20, 37890 unique headlines, 26 req/min
|
||||
3500 requests, window 2019-09-16, 38328 unique headlines, 26 req/min
|
||||
precise: 200 windows of 14 days x 21 queries; 6125 done, 523 to go
|
||||
100 requests, window 2024-03-25, 44823 unique headlines, 27 req/min
|
||||
200 requests, window 2021-10-11, 45308 unique headlines, 27 req/min
|
||||
@@ -0,0 +1,53 @@
|
||||
DeepSeek deepseek-v4-flash, prompt d32e04c6; budget 200 calls
|
||||
pools: precise 2778, broad 20532 unlabelled; picked 5000
|
||||
selected 5000 unlabelled headlines
|
||||
250 labelled, 10 calls (0 failed), 0 from cache
|
||||
DeepSeek batch of 25 failed (attempt 1/3, None) — retrying
|
||||
DeepSeek batch of 25 failed (attempt 2/3, None) — retrying
|
||||
DeepSeek batch of 25 failed (attempt 3/3, None) — retrying
|
||||
475 labelled, 22 calls (3 failed), 0 from cache
|
||||
DeepSeek batch of 25 failed (attempt 1/3, None) — retrying
|
||||
DeepSeek batch of 25 failed (attempt 2/3, None) — retrying
|
||||
DeepSeek batch of 25 failed (attempt 3/3, None) — retrying
|
||||
700 labelled, 34 calls (6 failed), 0 from cache
|
||||
950 labelled, 44 calls (6 failed), 0 from cache
|
||||
1200 labelled, 54 calls (6 failed), 0 from cache
|
||||
DeepSeek batch of 25 failed (attempt 1/3, None) — retrying
|
||||
DeepSeek batch of 25 failed (attempt 2/3, None) — retrying
|
||||
DeepSeek batch of 25 failed (attempt 3/3, None) — retrying
|
||||
1425 labelled, 66 calls (9 failed), 0 from cache
|
||||
1675 labelled, 76 calls (9 failed), 0 from cache
|
||||
1925 labelled, 86 calls (9 failed), 0 from cache
|
||||
2175 labelled, 96 calls (9 failed), 0 from cache
|
||||
2425 labelled, 106 calls (9 failed), 0 from cache
|
||||
2675 labelled, 116 calls (9 failed), 0 from cache
|
||||
2925 labelled, 126 calls (9 failed), 0 from cache
|
||||
3175 labelled, 136 calls (9 failed), 0 from cache
|
||||
DeepSeek batch of 25 failed (attempt 1/3, None) — retrying
|
||||
3425 labelled, 147 calls (10 failed), 0 from cache
|
||||
3675 labelled, 157 calls (10 failed), 0 from cache
|
||||
3925 labelled, 167 calls (10 failed), 0 from cache
|
||||
4175 labelled, 177 calls (10 failed), 0 from cache
|
||||
4425 labelled, 187 calls (10 failed), 0 from cache
|
||||
4675 labelled, 197 calls (10 failed), 0 from cache
|
||||
4750 labelled, 200 calls (10 failed), 0 from cache
|
||||
this run: 200 calls, 10 failed, 4750 headlines labelled
|
||||
6250 labelled headlines
|
||||
year n bear irrel bull relevant mean xau
|
||||
2009 10 0 10 0 0.0% +0.000
|
||||
2010 9 0 9 0 0.0% +0.000
|
||||
2011 10 0 10 0 0.0% +0.000
|
||||
2012 2 0 2 0 0.0% +0.000
|
||||
2015 13 0 13 0 0.0% +0.000
|
||||
2016 6 0 6 0 0.0% +0.000
|
||||
2017 30 0 30 0 0.0% +0.000
|
||||
2018 19 0 19 0 0.0% +0.000
|
||||
2019 462 23 359 58 17.5% +0.040
|
||||
2020 507 29 366 69 19.3% +0.061
|
||||
2021 638 45 496 65 17.2% +0.018
|
||||
2022 707 72 477 97 23.9% +0.020
|
||||
2023 796 55 574 108 20.5% +0.037
|
||||
2024 807 48 568 139 23.2% +0.062
|
||||
2025 1067 97 543 329 39.9% +0.118
|
||||
2026 1167 218 559 258 40.8% +0.005
|
||||
(relevant = |xau| >= 0.25, irrel = |xau| <= 0.10, the rest is the grey zone)
|
||||
@@ -0,0 +1,44 @@
|
||||
DeepSeek deepseek-v4-flash, prompt d32e04c6; budget 200 calls
|
||||
pools: precise 6600, broad 0 unlabelled; picked 5000
|
||||
selected 5000 unlabelled headlines
|
||||
250 labelled, 10 calls (0 failed), 0 from cache
|
||||
500 labelled, 20 calls (0 failed), 0 from cache
|
||||
DeepSeek batch of 25 failed (attempt 1/3, None) — retrying
|
||||
750 labelled, 31 calls (1 failed), 0 from cache
|
||||
1000 labelled, 41 calls (1 failed), 0 from cache
|
||||
1250 labelled, 51 calls (1 failed), 0 from cache
|
||||
1500 labelled, 61 calls (1 failed), 0 from cache
|
||||
1750 labelled, 71 calls (1 failed), 0 from cache
|
||||
2000 labelled, 81 calls (1 failed), 0 from cache
|
||||
2250 labelled, 91 calls (1 failed), 0 from cache
|
||||
2500 labelled, 101 calls (1 failed), 0 from cache
|
||||
2750 labelled, 111 calls (1 failed), 0 from cache
|
||||
3000 labelled, 121 calls (1 failed), 0 from cache
|
||||
3250 labelled, 131 calls (1 failed), 0 from cache
|
||||
3500 labelled, 141 calls (1 failed), 0 from cache
|
||||
3750 labelled, 151 calls (1 failed), 0 from cache
|
||||
4000 labelled, 161 calls (1 failed), 0 from cache
|
||||
4250 labelled, 171 calls (1 failed), 0 from cache
|
||||
4500 labelled, 181 calls (1 failed), 0 from cache
|
||||
4750 labelled, 191 calls (1 failed), 0 from cache
|
||||
4975 labelled, 200 calls (1 failed), 0 from cache
|
||||
this run: 200 calls, 1 failed, 4975 headlines labelled
|
||||
14625 labelled headlines
|
||||
year n bear irrel bull relevant mean xau
|
||||
2009 10 0 10 0 0.0% +0.000
|
||||
2010 10 0 10 0 0.0% +0.000
|
||||
2011 10 0 10 0 0.0% +0.000
|
||||
2012 2 0 2 0 0.0% +0.000
|
||||
2015 13 0 13 0 0.0% +0.000
|
||||
2016 6 0 6 0 0.0% +0.000
|
||||
2017 33 0 33 0 0.0% +0.000
|
||||
2018 19 0 19 0 0.0% +0.000
|
||||
2019 1003 78 648 184 26.1% +0.056
|
||||
2020 1211 80 762 219 24.7% +0.064
|
||||
2021 1264 104 876 151 20.2% +0.018
|
||||
2022 1536 145 930 284 27.9% +0.045
|
||||
2023 1691 145 1082 295 26.0% +0.046
|
||||
2024 2047 132 1325 408 26.4% +0.071
|
||||
2025 3071 222 1671 829 34.2% +0.109
|
||||
2026 2699 345 1379 635 36.3% +0.047
|
||||
(relevant = |xau| >= 0.25, irrel = |xau| <= 0.10, the rest is the grey zone)
|
||||
@@ -0,0 +1,25 @@
|
||||
DeepSeek deepseek-v4-flash, prompt d32e04c6; budget 20 calls
|
||||
pools: precise 3137, broad 20533 unlabelled; picked 500
|
||||
selected 500 unlabelled headlines
|
||||
250 labelled, 10 calls (0 failed), 0 from cache
|
||||
500 labelled, 20 calls (0 failed), 0 from cache
|
||||
this run: 20 calls, 0 failed, 500 headlines labelled
|
||||
1500 labelled headlines
|
||||
year n bear irrel bull relevant mean xau
|
||||
2009 4 0 4 0 0.0% +0.000
|
||||
2010 6 0 6 0 0.0% +0.000
|
||||
2011 9 0 9 0 0.0% +0.000
|
||||
2012 2 0 2 0 0.0% +0.000
|
||||
2015 8 0 8 0 0.0% +0.000
|
||||
2016 5 0 5 0 0.0% +0.000
|
||||
2017 12 0 12 0 0.0% +0.000
|
||||
2018 10 0 10 0 0.0% +0.000
|
||||
2019 149 13 112 16 19.5% +0.022
|
||||
2020 157 10 117 19 18.5% +0.042
|
||||
2021 201 17 154 23 19.9% +0.020
|
||||
2022 188 17 129 29 24.5% +0.034
|
||||
2023 194 13 150 19 16.5% +0.018
|
||||
2024 226 11 164 40 22.6% +0.066
|
||||
2025 191 17 117 38 28.8% +0.062
|
||||
2026 138 22 71 28 36.2% +0.004
|
||||
(relevant = |xau| >= 0.25, irrel = |xau| <= 0.10, the rest is the grey zone)
|
||||
@@ -0,0 +1,49 @@
|
||||
DeepSeek deepseek-v4-flash, prompt d32e04c6; budget 140 calls
|
||||
pools: precise 3491, broad 18244 unlabelled; picked 3500
|
||||
selected 3500 unlabelled headlines
|
||||
250 labelled, 10 calls (0 failed), 0 from cache
|
||||
500 labelled, 20 calls (0 failed), 0 from cache
|
||||
750 labelled, 30 calls (0 failed), 0 from cache
|
||||
1000 labelled, 40 calls (0 failed), 0 from cache
|
||||
1250 labelled, 50 calls (0 failed), 0 from cache
|
||||
DeepSeek batch of 25 failed (attempt 1/3, None) — retrying
|
||||
DeepSeek batch of 25 failed (attempt 2/3, None) — retrying
|
||||
DeepSeek batch of 25 failed (attempt 3/3, None) — retrying
|
||||
1475 labelled, 62 calls (3 failed), 0 from cache
|
||||
1725 labelled, 72 calls (3 failed), 0 from cache
|
||||
1975 labelled, 82 calls (3 failed), 0 from cache
|
||||
WARNING llm_client: DeepSeek HTTP 502: <html>
|
||||
<head><title>502 Bad Gateway</title></head>
|
||||
<body>
|
||||
<center><h1>502 Bad Gateway</h1></center>
|
||||
<hr><center>nginx</center>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
DeepSeek batch of 25 failed (attempt 1/3, (1790538226.2151284, 'http_502')) — retrying
|
||||
2225 labelled, 93 calls (4 failed), 0 from cache
|
||||
2475 labelled, 103 calls (4 failed), 0 from cache
|
||||
2725 labelled, 113 calls (4 failed), 0 from cache
|
||||
2975 labelled, 123 calls (4 failed), 0 from cache
|
||||
3225 labelled, 133 calls (4 failed), 0 from cache
|
||||
3400 labelled, 140 calls (4 failed), 0 from cache
|
||||
this run: 140 calls, 4 failed, 3400 headlines labelled
|
||||
9650 labelled headlines
|
||||
year n bear irrel bull relevant mean xau
|
||||
2009 10 0 10 0 0.0% +0.000
|
||||
2010 10 0 10 0 0.0% +0.000
|
||||
2011 10 0 10 0 0.0% +0.000
|
||||
2012 2 0 2 0 0.0% +0.000
|
||||
2015 13 0 13 0 0.0% +0.000
|
||||
2016 6 0 6 0 0.0% +0.000
|
||||
2017 33 0 33 0 0.0% +0.000
|
||||
2018 19 0 19 0 0.0% +0.000
|
||||
2019 653 48 423 135 28.0% +0.073
|
||||
2020 701 58 438 139 28.1% +0.075
|
||||
2021 790 71 562 103 22.0% +0.025
|
||||
2022 1091 127 641 207 30.6% +0.037
|
||||
2023 1184 121 731 225 29.2% +0.045
|
||||
2024 1270 106 753 307 32.5% +0.086
|
||||
2025 2010 188 953 660 42.2% +0.130
|
||||
2026 1848 316 823 495 43.9% +0.042
|
||||
(relevant = |xau| >= 0.25, irrel = |xau| <= 0.10, the rest is the grey zone)
|
||||
@@ -0,0 +1,69 @@
|
||||
### small_lr5e5_e8 --epochs 8
|
||||
epoch 1: loss 1.6317 dev 3-class 0.680 rel F1 0.744 dir acc 0.630 thr 0.4 (3s)
|
||||
epoch 2: loss 0.9693 dev 3-class 0.755 rel F1 0.758 dir acc 0.852 thr 0.5 (6s)
|
||||
epoch 3: loss 0.5949 dev 3-class 0.735 rel F1 0.828 dir acc 0.852 thr 0.25 (9s)
|
||||
epoch 4: loss 0.3509 dev 3-class 0.745 rel F1 0.868 dir acc 0.778 thr 0.45 (11s)
|
||||
epoch 5: loss 0.2509 dev 3-class 0.755 rel F1 0.835 dir acc 0.870 thr 0.2 (14s)
|
||||
epoch 6: loss 0.1451 dev 3-class 0.755 rel F1 0.867 dir acc 0.889 thr 0.65 (16s)
|
||||
epoch 7: loss 0.1023 dev 3-class 0.760 rel F1 0.813 dir acc 0.889 thr 0.25 (19s)
|
||||
epoch 8: loss 0.0671 dev 3-class 0.750 rel F1 0.881 dir acc 0.870 thr 0.65 (21s)
|
||||
best epoch 7, relevance threshold 0.25
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.681 macro-F1 0.677 rel P/R/F1 0.721/0.876/0.791 dir acc 0.913
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep/small_lr5e5_e8/model.pt and meta.json
|
||||
### small_lr3e5_e6 --epochs 6 --lr 3e-5
|
||||
epoch 1: loss 1.6456 dev 3-class 0.695 rel F1 0.766 dir acc 0.685 thr 0.65 (3s)
|
||||
epoch 2: loss 1.0588 dev 3-class 0.745 rel F1 0.748 dir acc 0.815 thr 0.6 (6s)
|
||||
epoch 3: loss 0.7237 dev 3-class 0.740 rel F1 0.810 dir acc 0.852 thr 0.3 (8s)
|
||||
epoch 4: loss 0.5056 dev 3-class 0.765 rel F1 0.787 dir acc 0.833 thr 0.25 (11s)
|
||||
epoch 5: loss 0.3863 dev 3-class 0.770 rel F1 0.803 dir acc 0.870 thr 0.3 (14s)
|
||||
epoch 6: loss 0.3341 dev 3-class 0.750 rel F1 0.803 dir acc 0.870 thr 0.3 (17s)
|
||||
best epoch 5, relevance threshold 0.3
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.646 macro-F1 0.646 rel P/R/F1 0.675/0.889/0.767 dir acc 0.870
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep/small_lr3e5_e6/model.pt and meta.json
|
||||
### small_lr2e5_e8 --epochs 8 --lr 2e-5
|
||||
epoch 1: loss 1.7090 dev 3-class 0.680 rel F1 0.718 dir acc 0.685 thr 0.6 (3s)
|
||||
epoch 2: loss 1.2253 dev 3-class 0.720 rel F1 0.773 dir acc 0.759 thr 0.65 (6s)
|
||||
epoch 3: loss 0.8685 dev 3-class 0.740 rel F1 0.769 dir acc 0.852 thr 0.25 (9s)
|
||||
epoch 4: loss 0.6338 dev 3-class 0.760 rel F1 0.787 dir acc 0.833 thr 0.25 (11s)
|
||||
epoch 5: loss 0.4953 dev 3-class 0.760 rel F1 0.785 dir acc 0.870 thr 0.25 (14s)
|
||||
epoch 6: loss 0.4410 dev 3-class 0.745 rel F1 0.781 dir acc 0.852 thr 0.35 (17s)
|
||||
epoch 7: loss 0.3537 dev 3-class 0.760 rel F1 0.794 dir acc 0.870 thr 0.3 (19s)
|
||||
epoch 8: loss 0.3262 dev 3-class 0.755 rel F1 0.820 dir acc 0.870 thr 0.35 (22s)
|
||||
best epoch 4, relevance threshold 0.25
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.620 macro-F1 0.624 rel P/R/F1 0.638/0.897/0.746 dir acc 0.849
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep/small_lr2e5_e8/model.pt and meta.json
|
||||
### small_lr5e5_e4_s7 --epochs 4 --seed 7
|
||||
epoch 1: loss 1.5257 dev 3-class 0.625 rel F1 0.774 dir acc 0.615 thr 0.75 (3s)
|
||||
epoch 2: loss 1.0113 dev 3-class 0.685 rel F1 0.773 dir acc 0.785 thr 0.45 (6s)
|
||||
epoch 3: loss 0.6608 dev 3-class 0.710 rel F1 0.812 dir acc 0.846 thr 0.7 (9s)
|
||||
epoch 4: loss 0.4799 dev 3-class 0.720 rel F1 0.837 dir acc 0.862 thr 0.8 (12s)
|
||||
best epoch 4, relevance threshold 0.8
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.629 macro-F1 0.610 rel P/R/F1 0.818/0.759/0.787 dir acc 0.857
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep/small_lr5e5_e4_s7/model.pt and meta.json
|
||||
### small_lr5e5_e4_bs32 --epochs 4 --batch-size 32
|
||||
epoch 1: loss 1.4895 dev 3-class 0.730 rel F1 0.787 dir acc 0.833 thr 0.45 (4s)
|
||||
epoch 2: loss 0.7746 dev 3-class 0.755 rel F1 0.784 dir acc 0.870 thr 0.25 (9s)
|
||||
epoch 3: loss 0.4328 dev 3-class 0.760 rel F1 0.833 dir acc 0.833 thr 0.3 (13s)
|
||||
epoch 4: loss 0.2717 dev 3-class 0.780 rel F1 0.803 dir acc 0.870 thr 0.2 (17s)
|
||||
best epoch 4, relevance threshold 0.2
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.656 macro-F1 0.656 rel P/R/F1 0.683/0.894/0.774 dir acc 0.899
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep/small_lr5e5_e4_bs32/model.pt and meta.json
|
||||
### base_lr2e5_e4 --base microsoft/deberta-v3-base --epochs 4 --lr 2e-5 --batch-size 32
|
||||
epoch 1: loss 1.5986 dev 3-class 0.675 rel F1 0.775 dir acc 0.648 thr 0.75 (7s)
|
||||
epoch 2: loss 0.9843 dev 3-class 0.755 rel F1 0.778 dir acc 0.889 thr 0.45 (13s)
|
||||
epoch 3: loss 0.5717 dev 3-class 0.760 rel F1 0.797 dir acc 0.889 thr 0.4 (19s)
|
||||
epoch 4: loss 0.4177 dev 3-class 0.765 rel F1 0.800 dir acc 0.907 thr 0.45 (26s)
|
||||
best epoch 4, relevance threshold 0.45
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.638 macro-F1 0.634 rel P/R/F1 0.678/0.873/0.763 dir acc 0.886
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep/base_lr2e5_e4/model.pt and meta.json
|
||||
### base_lr3e5_e6 --base microsoft/deberta-v3-base --epochs 6 --lr 3e-5 --batch-size 32
|
||||
epoch 1: loss 1.5635 dev 3-class 0.705 rel F1 0.786 dir acc 0.741 thr 0.7 (6s)
|
||||
epoch 2: loss 0.9028 dev 3-class 0.775 rel F1 0.800 dir acc 0.926 thr 0.65 (12s)
|
||||
epoch 3: loss 0.4669 dev 3-class 0.770 rel F1 0.867 dir acc 0.870 thr 0.75 (19s)
|
||||
epoch 4: loss 0.3115 dev 3-class 0.775 rel F1 0.840 dir acc 0.889 thr 0.55 (24s)
|
||||
epoch 5: loss 0.1834 dev 3-class 0.780 rel F1 0.855 dir acc 0.889 thr 0.4 (30s)
|
||||
epoch 6: loss 0.1201 dev 3-class 0.785 rel F1 0.879 dir acc 0.907 thr 0.65 (36s)
|
||||
best epoch 6, relevance threshold 0.65
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.673 macro-F1 0.671 rel P/R/F1 0.750/0.857/0.800 dir acc 0.910
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep/base_lr3e5_e6/model.pt and meta.json
|
||||
### SWEEP DONE
|
||||
@@ -0,0 +1,69 @@
|
||||
### small_e4_bs32 --epochs 4 --batch-size 32
|
||||
8887 labelled rows -> 8870 unique headlines (0 golden-set duplicates dropped)
|
||||
train n= 7460 relevant 2332 irrelevant 4492 grey 636 bull 1290 bear 1042 {'history': 5940, 'newsdata': 1520}
|
||||
dev n= 392 relevant 119 irrelevant 245 grey 28 bull 72 bear 47 {'newsdata': 82, 'history': 310}
|
||||
val n= 1018 relevant 378 irrelevant 501 grey 139 bull 171 bear 207 {'newsdata': 1018}
|
||||
epoch 1: loss 1.0518 dev 3-class 0.786 rel F1 0.820 dir acc 0.882 thr 0.6 (12s)
|
||||
epoch 2: loss 0.6078 dev 3-class 0.811 rel F1 0.877 dir acc 0.899 thr 0.65 (24s)
|
||||
epoch 3: loss 0.3592 dev 3-class 0.816 rel F1 0.841 dir acc 0.908 thr 0.4 (35s)
|
||||
epoch 4: loss 0.2141 dev 3-class 0.824 rel F1 0.865 dir acc 0.916 thr 0.25 (47s)
|
||||
best epoch 4, relevance threshold 0.25
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.708 macro-F1 0.704 rel P/R/F1 0.788/0.876/0.830 dir acc 0.907
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep2/small_e4_bs32/model.pt and meta.json
|
||||
### small_e4_bs64 --epochs 4
|
||||
8887 labelled rows -> 8870 unique headlines (0 golden-set duplicates dropped)
|
||||
train n= 7460 relevant 2332 irrelevant 4492 grey 636 bull 1290 bear 1042 {'history': 5940, 'newsdata': 1520}
|
||||
dev n= 392 relevant 119 irrelevant 245 grey 28 bull 72 bear 47 {'newsdata': 82, 'history': 310}
|
||||
val n= 1018 relevant 378 irrelevant 501 grey 139 bull 171 bear 207 {'newsdata': 1018}
|
||||
epoch 1: loss 1.0966 dev 3-class 0.755 rel F1 0.783 dir acc 0.866 thr 0.35 (8s)
|
||||
epoch 2: loss 0.5993 dev 3-class 0.816 rel F1 0.876 dir acc 0.899 thr 0.75 (15s)
|
||||
epoch 3: loss 0.3881 dev 3-class 0.824 rel F1 0.869 dir acc 0.916 thr 0.6 (23s)
|
||||
epoch 4: loss 0.2570 dev 3-class 0.821 rel F1 0.861 dir acc 0.916 thr 0.35 (30s)
|
||||
best epoch 3, relevance threshold 0.6
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.720 macro-F1 0.715 rel P/R/F1 0.803/0.884/0.841 dir acc 0.942
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep2/small_e4_bs64/model.pt and meta.json
|
||||
### small_e6_bs32 --epochs 6 --batch-size 32
|
||||
8887 labelled rows -> 8870 unique headlines (0 golden-set duplicates dropped)
|
||||
train n= 7460 relevant 2332 irrelevant 4492 grey 636 bull 1290 bear 1042 {'history': 5940, 'newsdata': 1520}
|
||||
dev n= 392 relevant 119 irrelevant 245 grey 28 bull 72 bear 47 {'newsdata': 82, 'history': 310}
|
||||
val n= 1018 relevant 378 irrelevant 501 grey 139 bull 171 bear 207 {'newsdata': 1018}
|
||||
epoch 1: loss 1.0595 dev 3-class 0.786 rel F1 0.826 dir acc 0.832 thr 0.55 (12s)
|
||||
epoch 2: loss 0.6070 dev 3-class 0.811 rel F1 0.845 dir acc 0.908 thr 0.35 (24s)
|
||||
epoch 3: loss 0.3743 dev 3-class 0.811 rel F1 0.855 dir acc 0.891 thr 0.25 (35s)
|
||||
epoch 4: loss 0.2223 dev 3-class 0.814 rel F1 0.854 dir acc 0.891 thr 0.2 (47s)
|
||||
epoch 5: loss 0.1206 dev 3-class 0.814 rel F1 0.877 dir acc 0.908 thr 0.7 (59s)
|
||||
epoch 6: loss 0.0744 dev 3-class 0.811 rel F1 0.851 dir acc 0.908 thr 0.25 (70s)
|
||||
best epoch 4, relevance threshold 0.2
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.713 macro-F1 0.708 rel P/R/F1 0.811/0.876/0.842 dir acc 0.897
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep2/small_e6_bs32/model.pt and meta.json
|
||||
### small_e8_bs64 --epochs 8
|
||||
8887 labelled rows -> 8870 unique headlines (0 golden-set duplicates dropped)
|
||||
train n= 7460 relevant 2332 irrelevant 4492 grey 636 bull 1290 bear 1042 {'history': 5940, 'newsdata': 1520}
|
||||
dev n= 392 relevant 119 irrelevant 245 grey 28 bull 72 bear 47 {'newsdata': 82, 'history': 310}
|
||||
val n= 1018 relevant 378 irrelevant 501 grey 139 bull 171 bear 207 {'newsdata': 1018}
|
||||
epoch 1: loss 1.1464 dev 3-class 0.763 rel F1 0.787 dir acc 0.866 thr 0.45 (8s)
|
||||
epoch 2: loss 0.6293 dev 3-class 0.796 rel F1 0.809 dir acc 0.882 thr 0.55 (15s)
|
||||
epoch 3: loss 0.4127 dev 3-class 0.798 rel F1 0.838 dir acc 0.908 thr 0.7 (23s)
|
||||
epoch 4: loss 0.2944 dev 3-class 0.801 rel F1 0.863 dir acc 0.891 thr 0.25 (30s)
|
||||
epoch 5: loss 0.1823 dev 3-class 0.806 rel F1 0.838 dir acc 0.916 thr 0.25 (38s)
|
||||
epoch 6: loss 0.1072 dev 3-class 0.804 rel F1 0.847 dir acc 0.916 thr 0.2 (46s)
|
||||
epoch 7: loss 0.0660 dev 3-class 0.819 rel F1 0.873 dir acc 0.899 thr 0.5 (53s)
|
||||
epoch 8: loss 0.0513 dev 3-class 0.821 rel F1 0.858 dir acc 0.916 thr 0.4 (60s)
|
||||
best epoch 8, relevance threshold 0.4
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.709 macro-F1 0.704 rel P/R/F1 0.789/0.889/0.836 dir acc 0.905
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep2/small_e8_bs64/model.pt and meta.json
|
||||
### small_e6_bs32_lr3e5 --epochs 6 --batch-size 32 --lr 3e-5
|
||||
8887 labelled rows -> 8870 unique headlines (0 golden-set duplicates dropped)
|
||||
train n= 7460 relevant 2332 irrelevant 4492 grey 636 bull 1290 bear 1042 {'history': 5940, 'newsdata': 1520}
|
||||
dev n= 392 relevant 119 irrelevant 245 grey 28 bull 72 bear 47 {'newsdata': 82, 'history': 310}
|
||||
val n= 1018 relevant 378 irrelevant 501 grey 139 bull 171 bear 207 {'newsdata': 1018}
|
||||
epoch 1: loss 1.1073 dev 3-class 0.773 rel F1 0.803 dir acc 0.849 thr 0.6 (12s)
|
||||
epoch 2: loss 0.6466 dev 3-class 0.814 rel F1 0.824 dir acc 0.899 thr 0.35 (23s)
|
||||
epoch 3: loss 0.4186 dev 3-class 0.819 rel F1 0.856 dir acc 0.908 thr 0.55 (35s)
|
||||
epoch 4: loss 0.2727 dev 3-class 0.814 rel F1 0.855 dir acc 0.908 thr 0.25 (47s)
|
||||
epoch 5: loss 0.1877 dev 3-class 0.816 rel F1 0.835 dir acc 0.891 thr 0.2 (58s)
|
||||
epoch 6: loss 0.1338 dev 3-class 0.821 rel F1 0.849 dir acc 0.899 thr 0.2 (70s)
|
||||
best epoch 6, relevance threshold 0.2
|
||||
VAL (unseen 2026-09-18+, n=1018): 3-class 0.714 macro-F1 0.711 rel P/R/F1 0.779/0.897/0.834 dir acc 0.926
|
||||
wrote /tmp/claude-1000/-home-harwinder-Development-ML-NEWS/2ebe8a2c-6870-461a-bffa-60b46627e57b/scratchpad/sweep2/small_e6_bs32_lr3e5/model.pt and meta.json
|
||||
### SWEEP2 DONE
|
||||
@@ -0,0 +1,34 @@
|
||||
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
|
||||
WARNING huggingface_hub.utils._http: Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
|
||||
/home/harwinder/Development/ML-NEWS/.venv-train/lib64/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
|
||||
warnings.warn(
|
||||
3637 labelled rows -> 3620 unique headlines (0 golden-set duplicates dropped)
|
||||
splits:
|
||||
train n= 2402 relevant 757 irrelevant 1448 grey 197 bull 275 bear 482 {'history': 921, 'newsdata': 1481}
|
||||
dev n= 200 relevant 54 irrelevant 132 grey 14 bull 22 bear 32 {'newsdata': 121, 'history': 79}
|
||||
val n= 1018 relevant 378 irrelevant 501 grey 139 bull 171 bear 207 {'newsdata': 1018}
|
||||
|
||||
Loading weights: 0%| | 0/102 [00:00<?, ?it/s]
|
||||
Loading weights: 100%|██████████| 102/102 [00:00<00:00, 1282.76it/s]
|
||||
[transformers] [1mDebertaV2Model LOAD REPORT[0m from: microsoft/deberta-v3-small
|
||||
Key | Status | |
|
||||
----------------------------------------+------------+--+-
|
||||
mask_predictions.classifier.weight | UNEXPECTED | |
|
||||
lm_predictions.lm_head.LayerNorm.bias | UNEXPECTED | |
|
||||
lm_predictions.lm_head.dense.weight | UNEXPECTED | |
|
||||
mask_predictions.dense.weight | UNEXPECTED | |
|
||||
mask_predictions.LayerNorm.weight | UNEXPECTED | |
|
||||
mask_predictions.LayerNorm.bias | UNEXPECTED | |
|
||||
lm_predictions.lm_head.bias | UNEXPECTED | |
|
||||
lm_predictions.lm_head.dense.bias | UNEXPECTED | |
|
||||
mask_predictions.dense.bias | UNEXPECTED | |
|
||||
mask_predictions.classifier.bias | UNEXPECTED | |
|
||||
lm_predictions.lm_head.LayerNorm.weight | UNEXPECTED | |
|
||||
|
||||
Notes:
|
||||
- UNEXPECTED: can be ignored when loading from different task/architecture; not ok if you expect identical arch.
|
||||
epoch 1: loss 1.5514 dev 3-class 0.680 rel F1 0.780 dir acc 0.685 thr 0.3 (3s)
|
||||
epoch 2: loss 0.9117 dev 3-class 0.770 rel F1 0.787 dir acc 0.852 thr 0.55 (6s)
|
||||
epoch 3: loss 0.6081 dev 3-class 0.775 rel F1 0.813 dir acc 0.889 thr 0.2 (9s)
|
||||
epoch 4: loss 0.4231 dev 3-class 0.775 rel F1 0.813 dir acc 0.870 thr 0.25 (12s)
|
||||
best epoch 3, relevance threshold 0.2
|
||||
Reference in New Issue
Block a user