A taster round in quantitative finance. Six market systems mapped by correlation and regime, then a narrowing onto four sectors and the violent high-volume gap-ups worth trading — and, five separate times, an instrument that answered before the market did.
Python · pandas · NumPy · statsmodels · yfinance · Wilson score intervals · permutation and surrogate testing · thinkScript / thinkorswim · JSON run history · HTML generated from Python
Bars are the share of up-gaps that closed above their open. The dashed marker on each track is that sector's ordinary-day rate — what the stock does with no gap at all. Distance from the marker is the signal; distance from 50% is nothing.
I wanted to know what quantitative research feels like from the inside, so I picked a question small enough to finish: when a stock opens sharply higher on heavy volume — a supernova, in day-trading shorthand — does the move continue through the session or fade?
Four sectors, four identical sandboxes, deliberately unlike each other — small-cap biotech, small-cap AI, cannabis, and bitcoin miners. Alongside it, a Markov chain study labeling each day into a market state, and a small intraday backtest with a held-out half.
The plan was to learn the tools. What I learned was how confidently the tools lie.
A gap-up on ordinary volume is worse than an ordinary day. That is the sturdiest thing in the study, and it is the opposite of what I went looking for.
Continuation means the stock closed above where it opened. Measured against each sector's own ordinary-day rate — not against 50%, which none of these names reach:
| vs own baseline | Biotech | n | AI | n |
|---|---|---|---|---|
| <1.5× volume | −16.0 | 337 | −12.3 | 775 |
| 1.5–2× | −6.1 | 85 | +6.2 | 147 |
| 2–3× | +5.8 | 56 | +11.6 | 133 |
| 3–5× | +21.0 | 32 | +17.0 | 91 |
| 5×+ | +19.9 | 35 | +21.0 | 81 |
Percentage points against baselines of 47.8% (biotech) and 45.2% (AI). 1,773 gaps. Flat prints excluded.
Two universes with nothing in common land on the same shape and nearly the same size — roughly a 35-point swing from floor to ceiling. That agreement is the result.
But the two ends are not equally trustworthy. The bottom rows rest on hundreds of observations. The top rows rest on thirty-two, with confidence intervals thirty points wide. Split by year, biotech's numbers held from 2024 to 2026 while AI's quiet-gap fade decayed until, by 2026, it carried no information at all.
So the honest product is a filter, not a signal: heavy volume is the only condition under which the setup is worth looking at, and I have measured which gaps to ignore far better than I have measured which to take.
Noise reproduced my headline result almost exactly. The regimes were a property of the smoother.
The regime study measured its own indicator. I labeled days into states using a 20-day moving average, built transition matrices, and found trends persisting about ten days with 0.90 probability — far above a shuffled baseline. Then I ran the identical labeling on a synthetic random walk, where regimes cannot exist:
| Persistence | Real (UNP) | Random walk |
|---|---|---|
| Trend up | 0.90 | 0.85 |
| Trend down | 0.84 | 0.87 |
| Chop | 0.60 | 0.49 |
Price stays on one side of a slow smoother for weeks by construction. Shuffling the labels tests whether a sequence is ordered, not whether regimes are real.
The volume denominator included the day it
measured. A twenty-day average computed with the gap day inside it
understates a genuine 10× day as 6.9×. One call to
.shift(1) moved every threshold in the study.
A metric that quietly penalized cheap stocks. Counting a close-equals-open day as a fade sounds neutral. Flat prints are a function of tick size against share price: 25% of days for a stock at $0.11, none at all for one at $44. The measure was biased against exactly the names this strategy selects for.
An entire sector built on unusable data. A mean of +14.5% sitting beside a median of −0.5% was the only visible symptom. Following it turned up 66 corrupted rows — gaps of +989,900%, the same impossible value repeating, sub-penny rounding masquerading as price moves. Cleaning them cut the sector's apparent edge by sixty percent and it was withdrawn.
A report with nothing behind it. The exit-side study rendered a clean dashboard: twelve gap sessions, every peak in the morning, an average of 9% given back. Twelve different dates — and every row identical to the last decimal. It was a placeholder render, not a result. The one genuine session the tool found looked far less convincing than the fabricated twelve.
The first four mistakes were arithmetic producing confident output. This one is confident output with no arithmetic behind it — the failure that survives a code review, because the code isn't what's wrong.
Nothing in any pipeline flagged any of this. Every script ran clean and printed a confident answer. The check that caught each one was asking what the number would look like if there were nothing there.
A rail portfolio is not six positions. It is one position held six times.
Before narrowing to single stocks, I mapped six systems — rail and freight, AI compute, the coffee chain, the cocoa chain, grain, and irrigation — by daily return correlation, lead-lag structure, and a two-state Markov switching model fitted in statsmodels rather than labeled by hand.
| Correlation | Pair | r |
|---|---|---|
| Rail, tightest | UNP · NSC | +0.82 |
| AI compute | NVDA · TSM | +0.71 |
| Chocolate vs cocoa | HSY · CC=F | −0.19 |
| Rail vs grain | UNP · ZC=F | −0.07 |
Two years of daily returns, 501 sessions per name.
The negative one is the most legible. Hershey moves against the cocoa price, because cocoa is a cost to Hershey rather than a shared revenue driver — the pass-through relationship showing up cleanly in correlation alone.
The rail number is the one that changes how you'd hold a portfolio. At +0.82 and +0.77 between the majors, owning UNP, CSX and NSC is not diversification within a sector. It is concentration wearing a portfolio's clothes.
I had also written a prediction into the code before running it: if UNP correlates more with grain than with its own rail peers, the rail anchor is really a grain proxy. It isn't. UNP sits at +0.82 with NSC and between −0.07 and 0.00 with corn, soy and wheat. The hypothesis was clean, testable, and wrong.
Before checking, I wrote down why I thought biotech worked and cannabis didn't: biotech gaps for company-specific reasons — a trial readout, a filing — while a cannabis headline moves every name at once. If that were right, cannabis gap days should cluster on shared calendar dates and biotech's should scatter.
The first half held. At heavy volume, 85% of biotech's gaps happen in isolation against 42% of AI's — catalyst independence is real and measurable.
The second half inverted. Cannabis gap days clustered least of the four sectors, and where clustering did occur it made continuation better, not worse. In AI, gaps landing on shared dates continued 67% of the time against 59% for isolated ones. A sector-wide move is confirmation that something real happened, not contamination.
Writing the prediction down before testing it is what made the reversal legible. An untested intuition would have survived as a finding.
A trend-entry rule on 2-minute bars, two exit variants, costs at 20 basis points a side. The data was cut 60/40 — the rule was designed on the first half and scored on a second half it had never seen.
| Held-out half | Trades | Win rate | Expectancy | Max DD |
|---|---|---|---|---|
| A — exit on chop | 92 | 11% | −0.419% | −33.0% |
| B — scale and trail | 26 | 23% | −0.188% | −6.4% |
Both lose. Scaling out and trailing loses far less — the exit structure matters, but it cannot rescue a bad entry.
An 11% win rate with symmetric win and loss sizes is well below what random entry produces. The entry isn't uninformative, it's anti-predictive — buying after two up bars on 2-minute data is systematically buying local tops.
The exit side never got further than that. Yahoo's intraday history reaches back sixty days, which yielded exactly one qualifying gap session to study — which is why this page has a measured entry filter and no exit rule.
That is a more useful outcome than a flat result, and it only exists because the held-out half was written before the rule was. A rule that only wins on the data it was designed on is a fantasy, and the temptation to tune until the design half looks prettier is the entire reason the split has to be built first.
Replace the shuffled baseline with volatility-matched surrogate series and a block bootstrap that preserves short-range autocorrelation. Persistence above that null would be a genuine regime finding. The real numbers do run a few points above the random walk — the current control just can't see it.
Every result here is a base rate on a single session. There is no exit rule, no position sizing, and no profit and loss. A measured base rate is a filter, and calling it a strategy would be the next mistake in this series.
If a signal is reliably anti-predictive, its inverse is worth testing — though 40 basis points of round-trip cost is a high bar for a 2-minute mean reversion, and the honest expectation is that it doesn't clear.
Each study renders its own HTML from Python and appends the run to a dated JSON history, so results accumulate instead of overwriting. Code and full output are public.
| Skill | What I did with it |
|---|---|
| Market Structure | |
| Correlation as concentration | Mapped six market systems by daily return correlation and lead-lag structure. A thesis basket whose members correlate at +0.82 is one bet held several times — the same variable that makes the thesis coherent is what makes the portfolio undiversified. |
| Base rates before signals | Measured what each basket does on an ordinary day — 42 to 48% of sessions close above their open — before asking what it does after a gap. Two sectors that looked different turned out to be one effect sitting on two baselines. |
| Regime stability | Split every result by year instead of pooling. Biotech held at both ends across three years; AI's quiet-gap signal decayed until it matched its own baseline. A weaker stable result is worth more than a strong unstable one. |
| Catalyst independence | Measured what share of each sector's high-volume gaps occur in isolation versus alongside their peers, turning an intuition about company-specific news into a number: 85% for biotech, 42% for AI. |
| Quantitative Modeling | |
| Markov transition matrices | Labeled each session into a market state, counted state-to-state transitions, and derived expected regime duration from the diagonal. Gave catalyst gaps their own states rather than discarding them as outliers, so the memoryless property could be applied to what follows them. |
| Null construction | Learned the difference between a shuffled null and a surrogate one. A shuffle tests whether a sequence is ordered; a volatility-matched random walk tests whether the structure is real. The second null reproduced my entire regime result. |
| Interval discipline | Attached Wilson score intervals to every cell. Several results I was ready to publish rested on ten to thirty observations with intervals thirty points wide — wide enough that the sectors I thought differed were indistinguishable. |
| Regime inference | Compared hand-labeled states against Markov switching models with switching variance fitted in statsmodels, so calm and stressed states were estimated from returns rather than assigned by rule. |
| Research Engineering | |
| Held-out validation | Split the backtest 60/40 with transaction costs included and scored only the half the rule never saw, then published the losing result instead of tuning parameters until the design half looked better. |
| Parallel sandboxes | Ran four sectors as independent, identically configured experiments rather than one combined study. That structure is what made their disagreements interpretable and let a sector artifact be told apart from a real effect. |
| Data validation | Checked the data before analyzing it. A mean far from its median led to 66 corrupted rows, sub-penny rounding artifacts, and the withdrawal of a whole sector — and to a price-dependent bias in the outcome metric itself. |
| Reproducible runs | Each study generates its own HTML report from Python and appends results to a dated JSON history, so runs accumulate and stay comparable instead of overwriting each other. |