|
PineForge unreleased
Deterministic PineScript v6 backtest runtime — C ABI reference
|
Thanks for your interest. This document is the whole workflow, written for someone who has never seen this repository. Read it once before your first change; after that, the checklist near the end is the part you keep.
Community interaction follows the Code of Conduct. For licensing and third-party obligations (Eigen, optional benchmark tools), see LEGAL.md. If you are an AI agent working from a brief rather than a person reading a guide, read Contributing as an LLM instead — same ground, written as rules with the guard that enforces each one.
PineForge is a C++17 backtest and forward-execution engine with a C ABI. Its job is to be right: a strategy run through it produces the same trade list TradingView produces for the same PineScript on the same bars, trade for trade, and two runs of the same inputs produce identical bytes. The PineScript → C++ transpiler is a separate project (pineforge-codegen); this repository is the runtime that every compiled strategy links against, and also a kernel that runs strategies written directly in C++ or C with no PineScript anywhere.
The rule: the kernel must not know what TradingView is. The test is mechanical — if justifying the change needs the word "TradingView", it does not belong in the kernel. The kernel changes only for a generic capability with a recorded ruling; the amended rule-2 test records the generic mechanism and the opt-in choice. TradingView's own rules go in the adapter or in codegen. The boundary, its five other rules and the ruling table live in ADR 0001.
Two consequences that are easy to miss:
NativeRunSpec field adds its row to ADR 0001's ruling table: native-only (and then it needs a native example and a test), adapter-policy, or adapter-hook. scripts/check_native_feature_rulings.py fails until the table is true.| You want to… | It goes in | And you must also |
|---|---|---|
| Fix a TradingView-parity difference | src/source/ or src/compat/pine/ | add a replay test on recorded bars; run the parity gate |
| Add a generic broker/matching capability | the kernel (src/engine_*, src/native_*) | make it opt-in, add its ADR 0001 ruling row, add a kernel-only test, and add an examples/native/ host if the ruling is native-only |
| Add or change a TA class | the right ta_*.cpp partition + its declaration in <pineforge/ta.hpp> | add a unit test against a hand-computed series |
| Change what codegen may emit | the contract, not this repo's runtime | say so in the PR; the transpiler lives in pineforge-codegen-oss |
Add a runtime PF_API export | src/c_abi.cpp + include/pineforge/pineforge.h | update EXPECTED_RUNTIME check_c_abi_runtime.py:29 and its two declaration counts, the ctypes harnesses, the README symbol table and every page that states the export counts — all in the same commit |
| Add a C kernel-driving export | src/native_c_host.cpp + include/pineforge/native_c_api.h | add it to EXPECTED_NATIVE_C_API (and its two counts) in scripts/check_c_abi_runtime.py, update that header's COVERAGE block, and retire the 1.0 C boundary row the export closes; scripts/check_native_c_api_surface.py proves the block is exactly the host's public surface and fails on a closed gap |
| Document something | docs/pages/, README.md, this file | cite the tree by path:line; the anchor guard checks that the line still holds the symbol |
You need CMake ≥ 3.16 (3.20 for ctest --test-dir), a C++17 compiler whose standard library has floating-point std::to_chars (libstdc++ from GCC 11 or later, or libc++ 14 or later), Python 3 and Eigen 3.3+ (fetched automatically if absent). The corpus's 1m feed is a Git LFS object, so install Git LFS (git lfs install) before the submodule update; ci_preflight.py needs actionlint 1.7.12 and ShellCheck; the kernel profile of ci_verify.py also builds the native live runner, which needs SQLite3, libcurl ≥ 7.86 and OpenSSL. On Debian/Ubuntu also install tzdata-legacy (sudo apt-get install -y tzdata-legacy): noble split the backward zone links out of tzdata, and test_native_calendar resolves US/Eastern, Japan and GB. Linux hosts running test_run_json_jailed_nofile_e2e require strace with permission to trace child processes (ptrace).
The stale-test-binary trap.
cmake --build build --target pineforgerebuilds only the static library. Test executables are separate targets that statically link it, andctestrebuilds nothing — so after a library-only build,ctestruns stale binaries and can pass falsely. Always build all targets beforectestwhen runtime values changed.
A change is proven by a test that failed before it and passes after it. In that order, deliberately:
tests/, or a native host under examples/native/, or a corpus probe. Register it: TEST_SOURCES tests/CMakeLists.txt:8 for a unit, or PINEFORGE_NATIVE_EXAMPLES examples/native/CMakeLists.txt:16 plus a _pf_example_line_hello_kernel examples/native/CMakeLists.txt:101 for the row's own assertion.A test whose body is a seed row, a return 0 main, a disabled row or a TODO placeholder is not evidence. Neither is a test whose expected value you copied out of the run you are trying to justify.
Run these before you push. Each one refuses a specific way of being wrong.
What each gate refuses:
| Gate | Refuses |
|---|---|
check_c_abi_runtime.py | a PF_API runtime export added or removed without its inventory row |
check_native_c_api_surface.py | a public NativeStrategyHost member with no C spelling and no recorded reason; a 1.0 C boundary row whose gap closed, or that docs/pages/native-engine.md stopped listing |
check_native_feature_rulings.py | a NativeRunSpec field the adapter does not declare and the ADR does not rule |
check_kernel_residuals.py | a TradingView-shaped name reaching the kernel archive or its installed headers without an ADR 0001 row |
check_kernel_seam_rows.py | a kernel virtual source_* seam, or a kernel member or row field the source layer writes, that no ADR 0001 row names in its first cell |
check_native_cpp_versions.py, check_aggregate_cpp_versions.py | an internal C++ epoch moved without its consumers |
check_adapter_spec_shadowing.py | the adapter setting a kernel field it is ruled not to set |
check_twin_parity.py | a frozen test assertion rewritten instead of a behaviour change being argued |
check_rng_draw_order.py | two random draws in one call's arguments or one operator's operands, whose order the compiler picks (x86-64 GCC and AppleClang would test different batteries) |
check_doc_anchors.py | a path:line citation that no longer points at the symbol it claims |
| check_doc_lint.py | a stale epoch, a roadmap label or a "there is no … yet" claim the tree has falsified | | check_pine_to_native_coverage.py | a Pine builtin with no row on the migration page | | check_doc_reverts.py | a published sentence deleted, or older wording restored over newer, by a commit whose message does not name it (by the lane label or hash of the commit that wrote it, or by six of its words) | | the CTest row floors | a test row that vanished from a profile |
ci_verify.py counts the CTest rows that actually ran and fails below a floor. The floors' numbers live in one place, the PRIVATE_FIXTURE_PROFILE_FLOORS ci_verify.py:808 table: per profile, full is its overlay-present registered population and pinned is how many rows of tests/private_fixture_rows.txt that registry registers, so with the private fixtures absent the floor is full minus pinned; every other profile carries its declared Profile.min_tests. A deleted or silently skipped row is a failure, not a quieter run. When rows land — or genuinely leave — re-measure the profile's own overlay-present Lab run and edit that table in the same commit, saying by how much. --min-tests overrides the floor for a local experiment; never in a commit.
The validation corpus is a byte oracle, not a smoke test.
It checks the corpus out at the gitlink this repository records, refuses to start if a pinned input under validation/ is modified, runs the probes, and fails if any probe's engine_trades.csv no longer hashes to what scripts/corpus_parity_baseline.txt pins — naming the probe and printing the first differing rows — or if scripts/verify_corpus.py no longer prints its pinned tier headline.
"Moved" means any byte changed: a one-tick slippage difference moves a probe's baseline hash while the tier headline stays excellent=311. Both matter. The baseline, not the corpus's committed trades, is the oracle: the tapes pineforge-corpus committed were written by an older harness and no longer reproduce.
An excellent grade also requires distinct-entry identity parity wherever TradingView has two or more distinct, non-empty entry Signals at one exact time, price and direction. Compare engine incarnation counts, not fragment row counts. Use every TradingView entry key to resolve prices: an exact match keeps its own key, even if that key has only one Signal or none; a non-exact price must be within the strict entry tolerance of only one key to count there. Overlap with another key, missing identities and count mismatches refuse excellent. Fragment consolidation and all other gates remain unchanged. See Distinct-entry identity and the private fixture set coof_cascade_identity's README for the nearby-price regression and the extra-entry and ambiguity guards.
A refresh is deliberate and carries its evidence in the same commit:
On a pull request the subset runs as advisory CI; the full sweep runs nightly and on demand, and the maintainers' own parity verification posts the pineforge/parity status the merge gate requires. The details, including which 54 probes and why, are in docs/ci.md.
Sometimes a pinned number should change — a test's expected value, a floor, a baseline hash. Loosening a pin to make a red thing green is the single easiest way to destroy this repository's value, so the convention is explicit: the diff that changes a pin carries, on the line above it or in the commit message,
with the what naming the mechanism, not the symptom. No note, no merge. You will find the convention already in scripts/ci_verify.py and in the test suites; follow the spelling exactly so it stays greppable.
Two different promises:
<pineforge/pineforge.h>, <pineforge/native_c_api.h>) is append-only within a major version: fields and functions are added at the end, never reordered, removed or retyped. PATCH never touches it, MINOR appends, MAJOR breaks and needs maintainer signoff. The static_asserts in src/c_abi.cpp pin the layouts. If those asserts fail in your branch, do not "fix" them by changing the asserts — find what changed in the C++ representation.inline namespace engine_script_run_v… and the hash domains) guard link-time compatibility between the library and the strategies compiled against it. Moving one is a real event with its own procedure: read docs/pages/abi-stability.md before you do, and expect the version guards to fail until every consumer moves with you.std::filesystem, no <format> in the library (src/, include/; the optional runner uses std::filesystem). Yes to structured bindings, if constexpr, std::optional..clang-format is canonical.lower_snake_case functions and members, PascalCase types.BacktestEngine method goes in the engine_*.cpp partition that matches its concern, a file-local helper into an anonymous namespace, a genuinely cross-TU helper into engine_internal.hpp under pineforge::internal.printf left behind, no commented-out experiment, no test.skip.The coverage harness is opt-in — instrumentation slows the build and leaves profile side-files:
Outputs land in build-cov/coverage/: totals.txt (per-file percentages), uncovered.txt (same rows, lowest first) and per-file/<source>.txt (annotated listings). Honoured env vars: BUILD_DIR, COMPILER, JOBS, SKIP_BUILD=1, SKIP_TESTS=1, FORMAT=html. A PR touching a file under 80 % should not lower that file's line coverage.
Read git log --format=s -40 before your first commit; the convention is visible and consistent:
<Area> is Docs, Tests, Kernel, Examples, Rulings, Integration, or the subsystem. The summary names the mechanism, not the symptom — "the CTest
row floor counts rows that ran; skipped rows are listed, not counted" rather than "fix floor". The body explains why, with the measurement. One concept per pull request; several commits inside it is fine.
A pull request body has two parts and no third:
ci_verify summary lines for both profiles, the parity result, the floor numbers if they moved. Paste it; do not summarise it.The merge gate is two commit statuses on the PR head, pineforge/verify and pineforge/parity, which the maintainers post after running the full ci_verify.py profiles and their parity verification on it. The GitHub Actions jobs (Linux x86_64 and arm64, Release and Debug; sanitizers, ctest, the source guards, the install/find_package smoke test and the parity subset) are advisory. A change that touches documentation only runs the preflight guards and the documentation build and skips the rest; docs/ci.md names the rule.
For changes measured against the closed TradingView test set, a sweep is run before and after over a fixed population, and two rules must both hold. On the hard surface (ETHUSDT.P at 15 minutes) no probe may regress: none may lose a grade, stop reporting a metric or become an engine error. Every other market and timeframe is pooled and scored over two bands, excellent and excellent+strong: +1 each time a probe enters a band, −1 each time one leaves it. A score above 0 passes, as long as no probe that leaves a band falls more than one grade. A score of exactly 0 passes only if no probe lost a grade or became an engine error and no hard-surface probe got worse on a graded metric; it is flagged no-improve-no-regression, or improved when a probe got better where the bands cannot see it (a grade rising below them or on the hard surface, or an engine error fixed). Any other 0 fails, and so does a negative score.
The most valuable thing you can send is a reproduction: the Pine script, the OHLCV slice, and TradingView's own trade list exported at full precision. That is exactly how every rule in this engine was found.
By contributing, you agree your contributions will be licensed under the Apache License 2.0 (the same license as the rest of the project). See [LICENSE](LICENSE).
corpus/ and benchmarks/assets to the intended commits and verify both commits are published on the public upstream repositories (pineforge-benchmarks-assets has no tags; its LEGAL.md states Apache-2.0). LEGAL.md describes the submodule split..env, or machine-specific paths in tracked files; keep benchmarks/_workdir, .venv and node_modules untracked.libpineforge (e.g. Eigen). Update LEGAL.md if you add a new mandatory runtime dependency.benchmarks/ tooling installs AGPL-covered PineTS. Default CI installs none of the benchmark tooling, so a minimal clone never pulls AGPL into the library build..github/workflows/release.yml (Actions → Release → Run workflow); rehearse with dry_run first. A dry run computes the version with scripts/release_version.py, builds, installs, verifies and packs every prebuilt tarball, and commits, tags, releases, uploads and dispatches nothing. A release candidate is override X.Y.Z-rc.N: a GitHub prerelease, dispatched to the hub with prerelease: true. The final release is override X.Y.Z (a candidate has no bump). From 1.0.0 the engine and codegen release one version number, prerelease included; see the public contract.