Pysolate

A Semantics-Aware Sandbox
for Efficient Isolated Execution

Yuzhe Shi
MSc Individual Project · Imperial College London

Running students’ code

Imagine a data analysis class. Students need a platform to run their code.

We start with a Python server. It executes submissions in the same interpreter.

student_01.pystudent_02.pystudent_03.py

Python server

exec(submission)

Read the scores, then take their mean

Normal runMalloryBob
import numpy as npscores = np.loadtxt("scores.csv")[1., 2., 3.]result = scores.mean()2.0

Mallory prepares a replacement function

Normal runMalloryBob
import numpy as np def missing_file(path):    raise FileNotFoundError(path) np.loadtxt = missing_file

Mallory changes the shared interpreter

Normal runMalloryBob
import numpy as np def missing_file(path):    raise FileNotFoundError(path) np.loadtxt = missing_file

Bob runs the original program

Normal runMalloryBob
import numpy as npscores = np.loadtxt("scores.csv")FileNotFoundErrorresult = scores.mean()

Bob had never met Mallory. His interpreter had.

Normal runMalloryBob
import numpy as npscores = np.loadtxt("scores.csv")FileNotFoundErrorresult = scores.mean()

Bob had never met Mallory. His interpreter had.

Normal runMalloryBob
import numpy as npscores = np.loadtxt("scores.csv")FileNotFoundErrorresult = scores.mean()

How can every run start clean?

Give each run its own interpreter

MalloryPython interpreternp.loadtxt is replacedBobPython interpreterResult: 2.0Each run has its own state.

Each interpreter repeats the preparation

MalloryPython interpreternp.loadtxt is replacedBobPython interpreterResult: 2.0Initialise Python → Import NumPyPrepared memoryABCInitialise Python → Import NumPyPrepared memoryABC…

Each interpreter repeats the preparation

MalloryPython interpreternp.loadtxt is replacedBobPython interpreterResult: 2.0Initialise Python → Import NumPyPrepared memoryABCInitialise Python → Import NumPyPrepared memoryABC…

Can we keep runs independent and use fewer resources?

Keep runs independent. Reorganise memory and tool work.

The task keeps its dependenciesand required results.Work and state can bemanaged separately.Choose when work runsand where state is kept.ProcessDecompose / organiseComputeToolI/O

Two optimizations built on this idea

Share prepared memoryPrepare Python once. Dashed blocksshare that state. Solid blocks holdeach run’s own changes.

Two optimizations built on this idea

Share prepared memoryPrepare Python once. Dashed blocksshare that state. Solid blocks holdeach run’s own changes.Start tool work earlierStart tool work before the programneeds its result. Independentoperations can overlap.One after anotherOverlapping workTime

Separate interpreters, shared starting pages

MalloryPython interpreterReady to runBobPython interpreterReady to runInitial memory imageABCPrepare the starting contents once.

The Host can access the interpreter’s memory

Go HostWasm instanceCPythonscores = np.loadtxt("scores.csv")result = scores.mean()Linear memoryPython objects and modulesHost readsand saves bytesCPython is compiledto WebAssembly (Wasm).Submitted codestays ordinary Python.

Load the CPython Wasm module

Host actions

1Load CPython module
2runtime_init
3runtime_prepare
4Snapshot

CPython memory

+ Module data

+ Python objects

+ NumPy objects

Saved linear memory

Initialise Python inside the instance

Host actions

1Load CPython module
2runtime_init
3runtime_prepare
4Snapshot

CPython memory

+ Module data

+ Python objects

+ NumPy objects

Saved linear memory

Warm the interpreter with NumPy

Host actions

1Load CPython module
2runtime_init
3runtime_prepare
4Snapshot

CPython memory

+ Module data

+ Python objects

+ NumPy objects

Saved linear memory

Capture the warmed interpreter state

Host actions

1Load CPython module
2runtime_init
3runtime_prepare
4Snapshot

CPython memory

+ Module data

+ Python objects

+ NumPy objects

Saved linear memory

The Host reads linear memory as []byte. After setup returns, we can save these bytes and restore them into a fresh instance.

Both interpreters reference the image

MalloryPython interpreterReady to runBobPython interpreterReady to runInitial memory imageABC

Both interpreters reference the image

MalloryPython interpreterAbout to write BBobPython interpreterStill uses BInitial memory imageABCWrite B

What happens when Mallory writes?

A write creates a private page

MalloryPython interpreterPrivate page readyBobPython interpreterStill uses BInitial memory imageABCB′Copy B

The interpreters keep their own changes

MalloryPython interpreternp.loadtxt is replacedBobPython interpreterResult: 2.0Initial memory imageABCB′Mallory’s changesBob still reads B

Discard private changes to reset the instance

MalloryPython interpreterReady for the next runBobPython interpreterResult: 2.0Initial memory imageABCB′Discard private changes. Reuse the prepared state.

More instances, greater savings

image/svg+xml Matplotlib v3.10.6, https://matplotlib.org/ −40 −20 0 1 2 4 8 Instances +4.6 -2.9 -18.1 -48.4 End-to-end time (s) −600 −400 −200 0 +59.5 +42.7 -314.4 -534.0 Memory: PSS (MiB) COW − per-run initialisation Saved Added

Dots: medians. Bars: 25th–75th percentiles.

PSS: the Host process with all instances open.

Faster reset for an answer-checking service

image/svg+xml Matplotlib v3.10.6, https://matplotlib.org/ 0 50 100 150 Median request latency (ms) Full-copy reset COW reset 154.5 ms 1.22 ms

Independent runs can share their starting pages

One interpreterPythonShared changesBob failsSeparate interpretersMalloryPythonBobPythonMemoryMemoryBob: 2.0Separate interpretersMalloryPythonBobPythonShared starting pagesBob: 2.0

From memory to external work

The task keeps its dependenciesand required results.Work and state can bemanaged separately.Choose when work runsand where state is kept.ProcessDecompose / organiseComputeToolI/O

What happens while a task waits for a tool?

Two price requests make Python wait

acme = market.recent_price("ACME")beta = market.recent_price("BETA")portfolio_value = acme + beta

External request

External request

Local calculation

The Host receives named tool requests

Go HostWasm instanceCPythonacme = market.recent_price( "ACME")beta = market.recent_price( "BETA")Linear memoryPython objects and modulesWASIWorkspacescores.csvTool ABIPrice toolHost handlerMarketservice

The sum needs both prices

acme = market.recent_price("ACME")beta = market.recent_price("BETA")portfolio_value = acme + betaACME readBETA readPython sum

Ordinary execution waits at each call

acme = market.recent_price("ACME")beta = market.recent_price("BETA")portfolio_value = acme + betaACME readBETA readPython sumExecutionWait for ACMEWait for BETASum

Ordinary execution waits at each call

acme = market.recent_price("ACME")beta = market.recent_price("BETA")portfolio_value = acme + betaACME readBETA readPython sumExecutionWait for ACMEWait for BETASum

Both tickers are known. Why wait?

Start the request before Python needs its result

price = market.recent_price("ACME")EarlierWhen Python reaches the callPrepareStart the requestReturn a handle immediatelyPython needs the result here

Return the result at the original call

price = market.recent_price("ACME")EarlierWhen Python reaches the callPrepareStart the requestReturn a handle immediatelyMaterializeReturn the valueor error

Start both price reads early

q_acme = prepare(market.recent_price, "ACME")q_beta = prepare(market.recent_price, "BETA") acme = materialize(q_acme)beta = materialize(q_beta)portfolio_value = acme + beta

Our ABI identifies the tool and its arguments.

Prepare is non-blocking. It starts the request and returns a handle.

PythonACME readBETA read200 ms200 msPrepare returns handles immediately0200 ms400 ms

Materialize delivers the result at the original call

q_acme = prepare(market.recent_price, "ACME")q_beta = prepare(market.recent_price, "BETA") acme = materialize(q_acme)beta = materialize(q_beta)portfolio_value = acme + beta

The first materialize blocks until ACME is ready. Python then reaches the second materialize. BETA is already ready, so it returns immediately.

PythonACME readBETA read200 ms200 msBlocked in first materializeFirst materialize returns.Then the second runs and returns immediately.0200 ms400 ms

The quote arrives while Python is still computing

q_acme = prepare(market.recent_price, "ACME")q_beta = prepare(market.recent_price, "BETA")heavy_work()acme = materialize(q_acme)beta = materialize(q_beta)portfolio_value = acme + beta

Suppose independent work runs before the price call. The quote arrives while Python is still busy.

Quote10:00ObservedPythonIndependent heavy workCall is still ahead

The quote arrives while Python is still computing

q_acme = prepare(market.recent_price, "ACME")q_beta = prepare(market.recent_price, "BETA")heavy_work()acme = materialize(q_acme)beta = materialize(q_beta)portfolio_value = acme + beta

Suppose independent work runs before the price call. The quote arrives while Python is still busy.

Quote10:00ObservedPythonIndependent heavy workCall is still ahead

Can we still use the quote at the call?

Heavy work can make the prepared quote too old

q_acme = prepare(market.recent_price, "ACME")q_beta = prepare(market.recent_price, "BETA")heavy_work()acme = materialize(q_acme)beta = materialize(q_beta)portfolio_value = acme + beta

The tool allows quotes up to five minutes old. By the time this call runs, our saved quote is too old.

Quote10:00ObservedPythonHeavy work takes 6 minutes10:06Original call siteQuote exceeds the 5-minute limit

Check the quote when Python reaches the call

q_acme = prepare(market.recent_price, "ACME")q_beta = prepare(market.recent_price, "BETA")heavy_work()acme = materialize(linearize(q_acme))beta = materialize(linearize(q_beta))portfolio_value = acme + beta

At the original call site, use the saved quote if it is for the same ticker and at most five minutes old. Otherwise, request a new quote.

Prepare, Linearize, Materialize: PLM.

Quote10:00ObservedPythonHeavy work takes 6 minutes10:06Original call siteFresh request

Prepare, Linearize, Materialize

price = market.recent_price("ACME")EarlierWhen Python reaches the callPrepareStart the requestReturn a handle immediatelyLinearizeCheck inputsand quote ageMaterializeReturn the valueor error

The programmer writes ordinary Python

Original Python

acme = market.recent_price("ACME")beta = market.recent_price("BETA")portfolio_value = acme + beta

Pysolate adds the steps

q_acme = prepare(    market.recent_price, "ACME")q_beta = prepare(    market.recent_price, "BETA")acme = materialize(linearize(q_acme))beta = materialize(linearize(q_beta))portfolio_value = acme + beta

Pysolate adds the steps automatically

Original Python

acme = market.recent_price("ACME")beta = market.recent_price("BETA")portfolio_value = acme + beta

Pysolate adds the steps

q_acme = prepare(    market.recent_price, "ACME")q_beta = prepare(    market.recent_price, "BETA")acme = materialize(linearize(q_acme))beta = materialize(linearize(q_beta))portfolio_value = acme + beta

Start tool requests as their calls arrive

Code arrives a piece at a time

Requests for this run

acme = market.recent_price("ACME")Readybeta = market.recent_price("BETA")In progressportfolio_value = acme + beta

… more code is arriving

Once the inputs are known, Pysolate can start these price reads.

The original program runs for the first time

User code

Requests for this run

acme = market.recent_price("ACME")Cache hitbeta = market.recent_price("BETA")Waitingportfolio_value = acme + beta

Python uses the saved ACME quote, then waits for BETA.

BETA returns, then Python continues

User code

Requests for this run

acme = market.recent_price("ACME")Cache hitbeta = market.recent_price("BETA")Returnedportfolio_value = acme + beta

BETA returns from the same prepared request. Python continues with the sum.

How much waiting can source generation hide?

One 1.5 s tool read overlapped a 1.4 s source stream.

End-to-end time fell by 22.3% compared with serial execution.

Total time: about 6.3 s → 4.9 s.

The 1.4 s saving is close to one full tool read.

Rewriting takes time too

The full program is ready. Each read takes 200 ms.

Ordinary execution

Run the program

With PLM

Rewrite the calls
Then run the program

Does the optimization save time overall?

One read leaves little work to overlap

The full program is ready. Each read takes 200 ms.
End-to-end time

image/svg+xml Matplotlib v3.10.6, https://matplotlib.org/ −25 −20 −15 −10 −5 0 5 Change in total time vs ordinary execution (%) 1 2 4 8 Independent calls +3.3% Saved Added

Dots: medians. Bars: 25th–75th percentiles.

More independent reads save more time

The full program is ready. Each read takes 200 ms.
End-to-end time

image/svg+xml Matplotlib v3.10.6, https://matplotlib.org/ −25 −20 −15 −10 −5 0 5 Change in total time vs ordinary execution (%) 1 2 4 8 Independent calls +3.3% -1.0% -9.3% -22.3% Saved Added

Dots: medians. Bars: 25th–75th percentiles.

Waiting runs still have state

Run AOwn interpreterComputingRun BOwn interpreterWaiting for a toolRun COwn interpreterWaiting for approvalRAMPrivate pagesPage 1Page 2Page 3Page 1Page 2Page 3Page 1Page 2Page 3

Research direction

Run B’s private pages are still in RAM

Run BWaiting for a toolIts Python state is kept.Host request pendingRun B’s private pagesRAMDisk (swap)Page 1Page 2Page 3

What should stay in memory while a run waits?

Move some pages out while Run B waits

Run BWaiting for a toolIts Python state is kept.Host request pendingRun B’s private pagesRAMDisk (swap)Page 1Available RAMfor other runsPage 2Page 3Swap out

Research direction

Read pages back when Python needs them

Run BTool reply is readyIts Python state is kept.Python continuesRun B’s private pagesRAMDisk (swap)Page 1Page 2Page 3Read backReading pages back takes time.

Research direction

Independent runs, better use of memory and time

Run AOwn interpreterRun BOwn interpreterRun COwn interpreterEach run keeps its own state and results.COWShare prepared pages.Keep writes private.99.2% lower request latency in ShimmyPLMStart tool reads earlier.Check and return at the call.22.3% lower end-to-end time with streaming

Questions

What is shared, rebuilt and restored?

Saved linear memory

Python heap

Imported module objects

Prepared data

Built for each instance

An instance of the same module

Fresh globals and tables

Host resources and workspace

Map the sealed image

memfd / MAP_PRIVATE

Set up the module

_initialize

Restore the image bytes

MADV_DONTNEED

Guest pointers are offsets. The Host mapping can move.

How do Host and Guest exchange data?

alloc(n) → request_offsetwrite JSON bytes at request_offsetexecute(request_offset, n)response_offset → [u32 length][JSON]dealloc(request_offset)

Two call directions

  • Host → Guest: source and inputs enter as JSON bytes.
  • Guest → Host: tool requests use request offset/length and reply offset/capacity.
  • The Host writes reply bytes and returns their length.

Offsets are 32-bit positions in linear memory. The response frame begins with a little-endian 4-byte payload length.

How does execution reuse an early request?

One run in the Go HostAnalyser GuestFind known tool callsExecution GuestRun the full programPLM request tableCall, operation and argumentsPrepareFind and checkExternal provider

Pending
Wait at the call

Ready
Check and return

Cannot reuse
Call the tool now

Where can the source pass place Prepare?

User code

ticker = "ACME"audit()label = "Latest quote"price = market.recent_price(ticker)result = [label, price]

Rewritten Python

ticker = "ACME"audit()q = prepare(market.recent_price, ticker)label = "Latest quote"price = materialize(linearize(q))result = [label, price]

Prepare goes after the ticker is set and after audit().

Pysolate applies the optimization automatically

Original Python

listing = tools.listing("ACME")if listing["market_open"]:    ticker = listing["ticker"]    label = "Current price"    price = market.recent_price(ticker)

Rewritten Python

listing = tools.listing("ACME")if listing["market_open"]:    ticker = listing["ticker"]    q = prepare(market.recent_price, ticker)    label = "Current price"    price = materialize(linearize(q))

What does each timer include?

COWBuild stateCreate instancesRunCloseIncludedShimmyInstall evaluatorHandle requestCopy responseResetIncludedSource-streamPLMPrepare workersSource arrivesFinal executionCloseIncludedWhole-programPLMPrepare engineGuest setupRewrite and runDecode and closeIncluded

How does NumPy run inside the Guest?

CPython + NumPynative codestdlib + NumPyagent_runtime filesWasm reactorstatic native modulesembedded filesystem

How import finds the native code

PyImport_AppendInittab(    "numpy.core._multiarray_umath",    PyInit__multiarray_umath);
  • Register native initialisers before starting CPython.
  • Pack Python files with wasi-vfs.
  • Import uses registered built-ins and packaged source.

Which tools can start early?

Price lookup in this exampleEarly reads are allowed.Maximum quote age at the call: 5 minutesBefore PrepareHas this tool been enabled for early calls?At the original callSame ticker? Quote age ≤ 5 minutes?Start early or keep the ordinary callUse the quote or request a new oneThe developer chooses which tools can start early.Pysolate checks the saved result when Python reaches the call.

Why preserve syntax and exception order?

Check the full source

open("out.txt", "w").write("done")if (

A late syntax error stops Python before the file write.

Complete sourceSyntaxError

Deliver errors at the call

price = market.recent_price("ACME")write_receipt()

A delayed error can allow the write to happen first.

At the callErrorStop

Error raised laterWriteError

Dirty pages reduce how many instances fit

With 28 GiB memory available

image/svg+xml Matplotlib v3.10.6, https://matplotlib.org/ 0 5 10 25 50 Pages made private per instance (%) 0 5k 10k 15k Estimated instances 14.9k 6.7k 2.1k 240

Our function model

F(a, t, h)

a Arguments

t Call time

h Hidden state

An early result may become invalid over time.

Check whether we can still use it.

Here, h includes the market’s current data, which may change before the call.

Prepare

Start the request early.

Linearize

Check the inputs and the saved result at the call.

Materialize

Return the value or error at that call.

Python can wait for approval and continue

Python local state stays with this run.Python computesHost waitsPython continuesKeep a local valueApprove → dispatchUse that same valueReject or expiry → no dispatchThe real CPython test continues the same execution.

What do the paging tests show?

Memory mapping testPython execution testBefore wait96 MiBDuring wait0 MiBAfter resume96 MiBRAM occupied by the mapped pagesObjects kept their values.Python continued after the wait.The next run started clean.