Cost, counted both ways

5.1

The cost nobody invoices

Storage is the cost that shows up on an invoice. The larger one is not seeing.

Physical change is slow. Installing sensors, building up history and validating a model takes months or years. Decisions now move faster than that: a new product line, a supplier changing, a process running hotter than it was designed for. When the question arrives, only what was already being observed can answer it.

That turns the usual cost question around. The question is not how little to keep, but what each stream is for. Different jobs need different data, and they do not all need everything.

5.2

Three jobs

Diagnosis looks back at a moment. Discovery looks for what recurs.

The jobs differ in what they need from time, and that decides how much of each stream has to be kept.

What each job needs
Job	Looks at	Needs	Can it be sampled?
Diagnosis	A past moment, when something failed	Full fidelity around the event, with enough confidence to act	No, around events
Discovery	Structure that recurs, including what nobody looked for	Coverage over time, without bias; it is also what models learn from	Yes, if the samples are unbiased
Live model	The present state of the system	Everything measured, continuously, with its outputs kept	Its outputs are kept; the raw signal need not be

5.3

Compressing what already exists

A narrow stream can be kept at a quarter of the size, at the same accuracy.

Most existing sensors sit close to one machine and report one variable. They capture little beyond it, so their value is in the events they see, not the quiet between them. Adaptive capture records slowly while the system is steady and at full rate while something is happening, much as a camera can record a frame or two a second and switch to twenty when something moves.

Same signal, same accuracy

Simulation Fig. 5.1 Compressing an existing, narrow stream. Same signal, same accuracy. Both records are read back by joining readings with straight lines, and both stay within 0.07 of the true signal. A fixed clock fast enough for the bursts runs at 56 readings a second throughout: 673 readings, 1.35 kB at 16 bits. Adaptive capture records twice a second while quiet and at 56 a second while something is happening, decided by watching the live signal at full rate and keeping 0.3 s either side of each event (shaded). About 150 readings, 0.33 kB including the switch points: four times less here, and more the longer the quiet. Related work: Florescu and Bhandari, Time Encoding via Unlimited Sampling. Simulated.

The live signal is watched at full rate. A stretch counts as active when its rate of change passes a threshold, and 0.3 s is kept either side so the start and end of each event are not lost. Inside active stretches the stream is kept at full rate; outside, twice a second.

Both records are read back the same way, by joining readings with straight lines, so the comparison is fair. Storage is 2 bytes a reading plus 8 bytes for each switch point. A simpler scheme that only records when the value moves by a set step looked better at first, but lost once both were read back with straight lines.

5.4

Keeping what a sensor would clip

A sensor that saturates loses the event at any storage price.

Diagnosis needs the large events most, and those are the ones a conventional converter cuts off at its limit. A folding converter wraps the signal back into its range instead, and the folds can be undone afterwards. The same hardware keeps events many times larger than its range.

One event, two sensors with the same range

Simulation Fig. 5.2 Same sensor range, two designs. A conventional converter clips at its limit, and the shape of the event above it is gone. A folding converter wraps the signal back into its range, like a clock passing twelve. For a band-limited signal sampled faster than a set rate (T ≤ 1/2Ωe), the folds can be undone exactly. Here the blue trace is rebuilt from the folded samples alone with the published recovery algorithm: take higher-order differences, which stay inside the range, fold them, then sum back up, rounding to whole folds at each step. In hardware tests, Bhandari's group at Imperial has recovered signals more than twenty times the converter's range. After Bhandari, Krahmer and Raskar, On Unlimited Sampling and Reconstruction, and Bhandari, Krahmer and Poskitt, Unlimited Sampling from Theory to Practice. Simulated.

Condition. The signal is band-limited to Ω and sampled with period T ≤ 1/(2Ωe). Then each higher-order difference of the samples is smaller than the one before.
Order. Take N = ⌈(ln λ − ln β) / ln(TΩe)⌉ differences, where λ is the converter's range and β bounds the signal. At that order the differences sit inside the range.
Unfold. Fold the N-th differences of the readings, which recovers the N-th differences of the true signal, then sum back up N times, rounding to whole multiples of 2λ at each step.
Result. In the simulation above the reconstruction matches the true signal to about 10⁻¹⁶, with N of 2 or 3.

5.5

Spending the savings

The budget freed by compression pays for sensors placed to see the rest.

A few high-sensitivity sensors, placed to overlap, carry most of the information a site produces. They measure continuously into a rolling buffer. What is stored follows one randomised schedule shared by all of them, so the overlap stays intact, and anything unusual in a pause is kept from the buffer. Total storage ends up comparable to today's.

One site, per day

Today

10.5 GB

Redesigned

9.7 GB

existing sensors, fixed ratediagnosis · same sensors, compressed · 0.33 GBdiscovery · three overlapping high-sensitivity sensors, stored half the time · 9.3 GB

What the overlapping sensors keep, one hour

Estimate Fig. 5.3 Where the storage goes. Today: four vibration sensors, one per machine (three-axis, 5 kHz), two thermal cameras and six environmental sensors, all at fixed rates. Redesigned: the same sensors compressed as in Table 5.1, plus three high-sensitivity three-axis sensors (12 kHz, 16-bit) placed to overlap. They measure all the time; what is stored follows one shared schedule of windows of random length, from 15 seconds to 10 minutes, each followed by an equal pause, so the three are always stored together. Random windows avoid falling into step with a machine's own cycle. When something unusual happens in a pause, the rolling buffer keeps it (teal); these stretches count toward the half. The random schedule avoids systematic gaps but cannot promise to catch every rare event; that is the buffer's job, and it is only as good as what it is told to treat as unusual. Derived markers from the live model add a few megabytes a day and are not shown. Simulated, from the rates in Table 5.1.

A fixed rhythm, such as a minute on and a minute off, can fall into step with a machine's own cycle and miss the same part of it every time. Windows of random length, from 15 seconds to 10 minutes, each followed by an equal pause, cannot. Each pause equals its window, so the share stored is exactly half over a day, though any single hour varies.

If the three sensors each followed their own schedule, all three would be stored together only about an eighth of the time, and the overlap would be lost. One shared schedule keeps them together. The buffer catches unusual events in the pauses, but only as well as the rule for what counts as unusual.

Modality	Example setting	Fixed rate, per day	Adaptive, per day
Environmental	Temperature, humidity, pressure, light at 1 Hz	1.4 MB	180 kBevery 20 s when steady · 4 Hz when changing fast · active 2% of the day
Gas sensor array	8 channels, 10 Hz, 32-bit	28 MB	2.7 MB0.5 Hz in clean air · 10 Hz on exposure · active 5%
Thermal image	160 × 120, 16-bit, one frame per minute	55 MB	25 MBa frame every 10 min when static · every 5 s when something moves · active 3%
Visible image	1080p JPEG, one frame per minute	≈ 0.7 GB	≈ 0.22 GBa frame every 5 min when static · every 5 s during events · active 1%
Pressure textile	16 × 16 grid, 20 Hz, 16-bit	0.9 GB	0.23 GB0.2 Hz when unoccupied · 20 Hz in use · active 25%
Vibration	3-axis, 5 kHz, 16-bit	2.6 GB	69 MB1 s at full rate each minute · continuous during anomalies · 1%
Acoustic	Mono, 16 kHz, 16-bit	2.8 GB	180 MB1 s at full rate each minute · continuous during events · 5%

Estimate Table 5.1 Indicative raw volumes, uncompressed except where noted. Real budgets depend on duty cycles, compression and retention tiers; lossless audio compression alone roughly halves the acoustic figure. The adaptive column records slowly while a system is steady and at full rate, or faster than today, while something is happening (Fig. 5.1); each figure follows from the rates and share of the day shown under it, which will differ from site to site.

5.6

Long volatility

Even capture is priced like an option.

A stock option costs a small premium now for the right to act later, and it is worth most when markets are volatile. Capturing evenly costs a little more than capturing for one question. What it buys is the right to answer other questions later, with methods that may not exist yet, and that right is worth most when conditions change fast. Capture built for one question is the opposite position: it pays only if the future looks like the present.

None of the proportions is fixed. Discovery data can be stored half the time, a quarter or an eighth; streams can move between fast and archive storage and back; sampling can be raised when a question appears.

5.7

Status and sources

Every figure in this note is a simulation of a published method or of an illustrative site, not a result from a deployment. The site in Fig. 5.3 is an example built from the rates in Table 5.1; real budgets depend on the site.

Bhandari, Krahmer and Raskar, On Unlimited Sampling and Reconstruction.
Bhandari, Krahmer and Poskitt, Unlimited Sampling from Theory to Practice.
Florescu and Bhandari, Time Encoding via Unlimited Sampling.

← Back to the manifesto, section 05