What a measurement can contain, what can be recovered from it, and where recovery stops.
Noise is not one thing.
In engineering, noise is whatever in a measurement is not the quantity the instrument was built to report. That is a definition about intent, not about physics. At least four different things sit under it, and they behave differently.
| Kind | What it is | What it carries | What can be done |
|---|---|---|---|
| Measurement noise | Thermal and electronic noise in the sensor itself | Nothing about the system | Averaged down, if independent (2.2) |
| Quantisation | Rounding to the converter's resolution | Nothing; it is bounded and small | Made negligible with resolution or dither |
| Interference | Other machines, flows, people, the structure | A signal about something else | Often separated and located (2.3, 2.6) |
| Deterministic complexity | Dynamics that look random but follow rules | The state of the system that produced it | Sometimes rebuilt from one series (2.5) |
Calibration and filtering act on all four at once. A filter set to keep a shaft's frequency removes thermal noise, but also every other process outside that band. Calibration fixes what counts as zero and what counts as the signal; the rest is discarded before anyone asks what it was.
This note asks which parts of that remainder can be recovered, by what methods and under what conditions, and where recovery is impossible however good the method.
A signal smaller than the noise can still be measured, if the noise does not repeat it.
Averaging N independent looks at the same signal leaves the signal unchanged and shrinks random noise by the square root of N. Correlating sensors, matched filtering and lock-in detection rest on the same principle: they accumulate what is consistent across observations and let what is inconsistent cancel.
The principle has a hard limit. If part of the noise is shared between observations, because sensors pick up the same interference or share a supply, averaging cannot remove that part. With a share ρ of the noise in common, the noise left after averaging N observations is σ√(ρ + (1 − ρ)/N), which never falls below σ√ρ however many sensors are added.
Each sensor's noise has variance σ², and any two share a correlation ρ. The variance of their mean is the sum of all N² covariances divided by N²: N variances σ² plus N(N − 1) covariances ρσ². That gives σ²(ρ + (1 − ρ)/N). As N grows, the second term vanishes and the first stays.
In the simulation each sensor's noise is built as √ρ times a common draw plus √(1 − ρ) times its own, which gives exactly that correlation. Placing sensors so they do not pick up the same interference, which is a question of geometry (2.6), is how ρ is brought down in practice.
Processes mixed in one signal can be told apart when they differ in form.
With one channel, separation relies on the sources being different in kind: different frequency bands, periodic against impulsive, smooth against sharp. Spectral filtering, matching pursuit and sparse decomposition all exploit differences of this sort, which is what Fig. 2.2 shows.
When two sources share the same form, nothing can separate them from one channel; the information is not in the measurement. A second channel changes that, if the sources reach the two channels in different proportions. Independent component analysis estimates those proportions from the data alone, and Comon showed it can do so, up to the order and scale of the sources, provided at most one of them is Gaussian. With fewer channels than sources, separation needs further assumptions, such as sparsity.
A sensor that saturates loses information no later processing can restore.
Clipping is the clearest case of information destroyed at acquisition. Unlimited sampling replaces the conventional converter with one that folds the signal back into its range. Bhandari, Krahmer and Raskar showed that a band-limited signal can be recovered from such folded samples when they are taken fast enough, with a sampling density that depends on the bandwidth only, not on how far the signal exceeds the range. The group has since demonstrated recovery well beyond a prototype converter's range in hardware.
The point here is general: the limit sat in the design of the measurement, not in the world, and changing the acquisition moved the boundary of what can be recovered. The full treatment, with the algorithm running live, is in the technical note Cost, counted both ways, section 5.4.
One time series can carry the state of the whole system that produced it.
Under conditions set out in the margin, Takens showed that delayed copies of a single observed variable, x(t), x(t − τ), x(t − 2τ) and so on, reconstruct the system's attractor faithfully for almost every choice of what is observed, provided enough copies are used. Sauer, Yorke and Casdagli extended this to attractors of fractal dimension: more than twice that dimension suffices. The shape of the hidden dynamics comes back from one measured coordinate.
This is the strongest reason not to discard a time series because it looks erratic. Whether it is noise or dynamics can be tested, and if it is dynamics, it carries more than the one variable it records.
Where sensors are placed decides what can be separated.
Two sensors see two sources in different mixtures depending on distance. Separating them means inverting that mixture. If both sensors see the sources in exactly the same proportion, the mixture cannot be inverted and the second sensor adds nothing. Close to that point it can still be inverted in principle, but any noise is amplified in what is recovered; the condition number measures by how much, and a large one means poor conditioning, not impossibility. Array signal processing builds localisation and beamforming on the same idea.
This is what overlap buys in the manifesto's Fig. 3: the same sensors, placed so each sees the site differently, make the mixture invertible. When the geometry is not known, blind methods estimate it, under the conditions in 2.3.
No method extracts information a measurement does not contain.
Two results bound everything above. The data processing inequality states that processing a measurement cannot increase the information it holds about its source. The Cramér–Rao bound states that no unbiased estimate of a quantity can be more precise than the measurement's Fisher information allows. Better algorithms move estimates towards these bounds; only better measurement moves the bounds themselves.
Averaging, separation, embedding and unfolding all recover information that was in the data. None creates it. The ceiling is set by what was measured, where, how often, with what range and what was kept, which is why the decisions that matter most are made at acquisition.
How should an observation system be designed when the questions, the phenomena and the methods are not yet known?
Every method above recovers information by trading on an assumption. Each one works when its assumption holds and fails, in the simulations above, when it does not. The assumption is usually made when the measurement is designed, which is when least is known.
| Method | What must already be assumed | What breaks it |
|---|---|---|
| Averaging | Something meaningful is shared, or can be aligned, across observations | Noise the sensors share (2.2) |
| Source separation | Sources differ in statistical or physical structure | Two or more Gaussian sources; sources of the same form (2.3) |
| Delay embedding | The record reflects low-dimensional deterministic dynamics, observed generically | Noise, short records, drift, an observation that hides a symmetry (2.5) |
| Sensor geometry | The mixing is identifiable from where the sensors sit | Sensors that see the same mixture (2.6) |
| Unlimited sampling | The signal is band-limited and sampled fast enough | Content beyond the band, or sampling too slow (2.4) |
Inference spends assumptions. Measurement for discovery has to keep possibilities open beyond the assumptions available today, because the methods that will read the record may not exist yet and the phenomena that matter may not have been named. That means designing for diversity of view, overlap, coverage in time, range and later reanalysis, without knowing exactly how each will be used.
A sensor can measure continuously without everything it measures being kept. A live computation can turn the stream into state estimates, statistics or learned representations, and keep those. Which raw readings are kept is a third decision. The separation opens a large design space, with one limit: a derived quantity answers the questions it was built for, and cannot be relied on for questions nobody anticipated. Raw readings, kept so that they represent the whole, remain the reserve for those.
Each simple rule for keeping raw readings fails in its own way. A fixed rate can fall into step with a periodic process and report it at the wrong frequency. Random windows are unbiased, but can miss a rare event and break relationships that span longer than a window. Adaptive capture keeps what its trigger recognises and sees the rest at a low fixed rate. Spreading the budget over more sensors adds views, but thins the record of each, and adds little where the views repeat one another. Shared schedules, rolling buffers, stratification and selection by uncertainty can each cover a weakness of another. The aim is not complete coverage, which no finite budget gives, but the most useful inference for what is spent, under whatever operational guarantees must hold.
Five ways of spending the same storage, judged on questions they were not designed for.
A simulated site has four sensors on a line and a machine at one end. Several things happen on it: a faint cycle in the machine, a few impacts, a high tone at the far end, a source somewhere in the middle that switches on and off, and a faint response somewhere else that follows each impact after a delay. Each acquisition strategy is given the same storage, a quarter of everything the four sensors measure, and designs what to keep without knowing what will be asked. Then the questions are revealed, and a fixed analysis, the same for every strategy, tries to answer each from what was kept.
The questions are hidden from the strategies, not unknown to us: we built the site and chose them. What is tested is robustness across a stated family of questions, not against any question whatever. Two strategies also watch the full live stream to decide what to keep; the comparison is of storage, and the cost of measuring and computing on everything is not counted.
impacts source switching on delayed responses
| Strategy | Storage used | 1 · Faint cycleDoes the machine have a cycle, and what is its period? | 2 · ImpactsHow many impacts, and when? | 3 · Far toneIs there a high tone at the far end, at what pitch? | 4 · LocateWhere is the source that switches on and off? | 5 · DelayDoes anything respond to the impacts, and after how long? |
|---|---|---|---|---|---|---|
| Fixed narrowOne sensor at the machine, at a fixed rate | 25% | 100% | 100% | 0% | 0% | 5% |
| Randomised windowsAll four sensors, each on its own random schedule | 25% | 25% | 0% | 85% | 30% | 0% |
| Overlapping geometryAll four sensors on one shared random schedule | 25% | 5% | 0% | 90% | 80% | 0% |
| AdaptiveFull rate around events each sensor flags, a fixed low rate otherwise | 25% | 15% | 100% | 5% | 100% | 30% |
| HybridEvents flagged anywhere keep all four sensors; the rest goes to shared random windows | 25% | 10% | 100% | 100% | 100% | 10% |
| Everything keptFor reference, at four times the budget | 100% | 100% | 100% | 100% | 100% | 95% |
The site. Four sensors at 0.08, 0.36, 0.64 and 0.92 along a line, each 12,000 readings long, with independent noise of unit variance. A source at distance d reaches a sensor with gain 1/(1 + (d/0.15)²). The machine at 0.08 has a cycle of amplitude 0.1, below the noise, with a period between 150 and 350 readings, and takes six impacts. A tone of amplitude 0.4 with a period of 3 to 8 readings sits at 0.92. A source between 0.3 and 0.7 switches on ten times for 150 readings. A response of amplitude 1 at 0.64 follows each impact after a delay of 600 to 2,500 readings.
The strategies. Each sees the budget, the sensor positions and, for the two that react to events, the live stream: they measure everything and decide what to keep. Event triggers fire on three readings beyond 3 within 20. The fixed rate, and the adaptive low rate, keep every k-th reading on a common clock. Random windows run from 20 to 600 readings. None sees the questions.
The analyses. Fixed in advance and blind to the strategy. Cycle and tone: a periodogram of the kept readings, which must show a peak above the level expected from noise alone, unique for the tone, within 2% of the true period. Impacts: all six found, none false. Location: the source's position fitted from the covariances of sensors kept at the same moments, within 5% of the site. Delay: the energy after each detected impact, against each sensor's quiet level, consistent across impacts, within 30 readings of the truth.
The check. The analyses were developed on the 20 worlds in the page, where with everything kept they answer every question except one delayed response. Frozen, and run on 200 fresh worlds, they do less well even with everything kept: impacts in 95% of worlds, the delay in 78% (section 2.10). Part of every failure in the table is the analysis, and the reference row says how much. The worlds are seeded, so every number can be reproduced; the code runs in this page and is published with the study.
The narrow sensor answers the two questions about its machine perfectly, because it spends the whole budget there, and nothing else. Spreading the same budget over four sensors buys the far tone and the location, and loses the faint cycle, which needed every reading at one place. A shared schedule locates the source far more often than independent ones, because location needs sensors kept at the same moments. Adaptive capture keeps every impact and locates the source, but its fixed low rate reports the far tone at an alias. The hybrid answers three of five reliably, and misses the faint cycle and the delay. The delayed response, a relationship across sensors and across time, defeats every strategy at this budget; at half the storage, the best, adaptive capture, manages about six worlds in ten.
Moving the storage slider shows where strategies cross over. Raising the budget does not help the narrow sensor, which has nowhere else to look.
The questions and the site are ours, and a different family of questions would rank the strategies differently: a site whose only question is the machine's cycle favours the narrow sensor outright. The analyses are deliberately simple. Storage is the only cost compared. Nothing here is a deployment. What the experiment supports is narrower than a ranking: under a fixed budget, each way of keeping data fails on a recognisable kind of question, and which kinds can be afforded to fail is a design decision that is better made explicitly.
The same experiment on 200 fresh worlds, under six changes of assumption.
Twenty worlds make a demonstration, not a test, and they are the worlds the analyses were developed on. So the analyses were frozen and the experiment rerun on 200 new worlds, first as it stands and then with one assumption changed at a time: a sensor that dies halfway through the day; a site that drifts; an event trigger set lower or higher; a different mix of phenomena; and stricter analyses. Each cell below counts the questions a strategy answered in at least 80% of worlds, out of five.
| Strategy | Baseline | Sensor fails halfway | Site drifts | Trigger 2.5 | Trigger 3.5 | Different mix | Stricter analyses |
|---|---|---|---|---|---|---|---|
| Fixed narrow | 2 | 1 | 0 | 2 | 2 | 1 | 2 |
| Randomised windows | 1 | 0 | 0 | 1 | 1 | 2 | 1 |
| Overlapping geometry | 1 | 0 | 0 | 1 | 1 | 2 | 1 |
| Adaptive | 2 | 1 | 1 | 2 | 2 | 2 | 1 |
| Hybrid | 3 | 2 | 0 | 2 | 3 | 3 | 3 |
| Everything kept | 4 | 3 | 2 | 4 | 4 | 3 | 4 |
| Strategy | 10% | 15% | 20% | 25% | 30% | 40% | 50% |
|---|---|---|---|---|---|---|---|
| Fixed narrow | 0 | 2 | 2 | 2 | 2 | 2 | 2 |
| Randomised windows | 0 | 0 | 0 | 1 | 1 | 1 | 2 |
| Overlapping geometry | 0 | 0 | 0 | 1 | 2 | 2 | 3 |
| Adaptive | 1 | 1 | 2 | 2 | 2 | 2 | 2 |
| Hybrid | 1 | 3 | 3 | 3 | 3 | 3 | 4 |
The narrow sensor kept its profile in every condition: it never answered a question away from its own machine, and more storage does not change that. A shared schedule located the source more often than independent ones in every condition, by a wide margin (80% of worlds against 28% at the baseline). No strategy answered the delayed response in more than 44% of worlds under any condition. The hybrid answered the most questions, alone or jointly, in six of the seven conditions, and at every storage level in the sweep, alone from 15% upward.
Adaptive capture's failure on the far tone depends on its trigger. Set lower, it triggers often enough at the far end to keep the tone at full rate, and finds it in 99% of worlds; the same lower trigger costs the hybrid its impacts (72%), because false events use up its budget. A failing sensor costs every strategy about what it costs the reference. When the site drifts, the fixed analyses themselves fail: the impact detector, set for the starting noise, raises false alarms in every world as the noise rises, and merges or misses some impacts, even with everything kept. Acquisition and analysis fail together, and a design judged under one analysis is not judged under all of them.
At the baseline, the fresh worlds broadly confirm the 20 in the page (a 95% interval is at most ±7 points at 200 worlds), with one larger difference: overlapping geometry found the faint cycle in 31% of fresh worlds, against 5% of the page's. The reference is weaker than it looked: with everything kept, the frozen analyses catch every impact in 95% of worlds and the delay in 78%. Every pass rate, with its interval, is in the published results.
Code and results: study notes · simulation · runner · results. Run with Node 18 or later, no dependencies.
These are the propositions we consider open and would test beyond simulation. They are hypotheses, not results.
For processes not chosen in advance, sensors placed to see a site in different mixtures preserve more recoverable structure per sensor than the same number placed together.
At equal storage, unbiased random retention of raw windows preserves more structure that can be recovered later than keeping derived features alone.
The share of noise an array's sensors have in common predicts what it can recover better than its sensor count.
For a stated family of future questions, a hybrid of event capture and shared random windows fails on fewer kinds of question than any single rule at the same storage. Fig. 2.5 and Tables 2.3 and 2.4 are consistent with this for one simulated site in six of seven conditions, and show its limits: relationships across time are not protected, and a drifting site defeats it.
How to estimate, before deployment, what a given acquisition design will make recoverable.
All simulations on this page run the stated models live in the browser. They show what the methods do under the stated assumptions, not results from a deployment.
UNDAMENTAL