Libraries Run Rust Inside Python (With PyO3)

Every time you validate data with Pydantic v2, the data-validation library most Python apps reach for, a Rust extension does the work. Its core, pydantic-core, is built with PyO3, the same toolchain we'll use here. This post builds that same kind of bridge, small enough to read in one sitting: a JSON parser written in Rust, exposed to Python, so you can import it like any other package. The last step, turning the Rust result into Python objects, is the one to understand before you port anything: for a parser like this, it can cost more than the parsing itself. The four steps from Rust to import Getting Rust code into Python takes four steps: Write a normal Rust module. Annotate it with PyO3 macros. Let maturin compile and install it. Import the result. #[pyfunction] and #[pymodule] are the two Rust macros that do the wiring. A Rust attribute macro is close to a Python decorator: it rewrites the function it sits on, here adding the glue that lets Python call it and handles the type conversions and reference counting at the boundary. Maturin then compiles the crate to a shared library (.so, .dylib, .dll) and drops it into your virtual environment, so import just works. I walk through this whole setup, from cargo new to the first import, in How to run Rust in Python with PyO3 and Maturin. That first tutorial returns a single number. This one picks up where it left off, because the interesting part starts once you return a structure instead of a scalar. The parser produces a Rust value first The structure this parser returns is a JSON tree, and it's the running example for the rest of this post. In our Python to Rust cohort, students spend six weeks writing a JSON parser from scratch in Rust, a hand-rolled tokenizer and recursive-descent parser with no serde, then expose it to Python through PyO3. Josh's version beat CPython's C json module on real-world fixtures; Jochen's ran up to 3.5x faster than the Python version. The public reference implementation, the clean version students start from, is the code I'll walk through here. The parser produces a plain Rust enum. A Rust enum holds one of several shapes, and each variant can carry data, so it maps a JSON tree cleanly: pub enum JsonValue { Null, Boolean(bool), Number(f64), String(String), Array(Vec ), Object(HashMap ), } That tree lives entirely in Rust. Python never sees it. The PyO3 layer is a thin adapter on top. Exposing one function Exposing a function to Python takes two lines: #[pyfunction] fn parse_json<'py>(py: Python<'py>, input: &str) -> PyResult <'py, PyAny>> { parse(input)?.into_pyobject(py) } For a Python reader, the signature is the most interesting part: py: Python<'py> is a token representing access to the Python interpreter and is what you pass to PyO3 APIs that need access to Python objects. On traditional Python builds, this access is associated with holding the GIL. PyO3 hands it to you and you pass it along wherever you touch a Python object. Bound<'py, PyAny> is a handle to a Python object of any type, the Rust side of what you'd think of as a PyObject. PyResult is Result : return the value, or an error PyO3 raises as a Python exception. ? propagates that error. If parse fails, the function returns early and Python sees an exception; otherwise it unwraps the JsonValue and moves on. So parse(input)? does the real work, and .into_pyobject(py) builds the Python objects the caller asked for. That last call is where the cost lives: it has to create Python objects for the nodes in the tree, and on a large document that can add up to more work than the parse itself. The return trip is the expensive part Here is why that conversion is not free. .into_pyobject walks the entire JsonValue tree and rebuilds it as native Python objects: a dict per object, a list per array, a float or str per leaf. You provide that translation by implementing the IntoPyObject trait, which PyO3 calls to convert a Rust value into a Python one: impl<'py> IntoPyObject<'py> for JsonValue { fn into_pyobject(self, py: Python<'py>) -> Result { match self { JsonValue::Null => Ok(py.None().into_bound(py)), JsonValue::Number(n) => Ok(n.into_pyobject(py)?.to_owned().into_any()), JsonValue::Object(obj) => { let py_dict = PyDict::new(py); for (k, v) in obj { py_dict.set_item(k, v.into_pyobject(py)?)?; // recurses } Ok(py_dict.into_any()) } // ...arrays, strings, booleans } } } A document with 100,000 values means on the order of 100,000 Python objects being created at the boundary, all after parsing is completely done. On a large document this materialization loop, not the parsing, can dominate the end-to-end time. Errors cross the boundary the same way The return value is not the only thing that has to translate. A parse failure is a typed Rust error, and Python wants an exception. One From impl, the trait Rust uses to convert one type into another, lets ? do the work: impl From for PyErr { fn from(err: JsonError) -> PyErr { match err { JsonError::UnterminatedString { position } => PyValueError::new_err( format!("Unterminated string starting at position {position}") ), // ...one arm per error variant, position preserved } } } Now malformed input raises a ValueError carrying the offset where parsing broke. The file-reading path gets the same treatment for free: std::io::Error already converts to the matching Python exception, so a missing path raises FileNotFoundError. The caller gets Python semantics without the Rust layer leaking through. What this means for your own port If the Rust function you're porting returns a scalar, port it and move on. The boundary is usually small enough to ignore. If it returns a large structure, the conversion is your real cost, and it is the next thing to optimize once the parser itself is fast. Preallocating the PyDict can help at the margins, but the bigger win is architectural: don't materialize the whole tree if the caller won't touch all of it. Hand back a lazy, Rust-backed view and build Python objects on demand. So when you reach for PyO3, profile the boundary, not just the algorithm. Getting Rust to run fast is the easy half. What you build on the way out, the trip from Rust values to Python objects, is the half that decides whether the port was worth it. Learning Rust? I co-run a 6-week Python to Rust cohort where you build a performant JSON parser with PyO3 bindings. Get my free guide What Developers Should Never Outsource to AI: three real case studies on using AI without giving up the judgment that makes you an engineer. Then emails on Python, Rust, and AI. Keep reading Guardrails Protect Your Codebase. What Protects Your Judgment? AI coding erodes two different things: your skills and your code. Guardrails protect the code. Only re-deriving the hard decisions keeps your judgment sharp. Learning New Skills in the AI Era (vBrownBag) I joined the vBrownBag podcast to talk about learning new languages and skills when AI can write the code before you finish the thought. Rust, AI, and the Developer Mindset (Develpreneur Podcast) I joined the Develpreneur podcast with Jim Hodapp to talk about the Rust developer mindset and why the compiler is a great guardrail for AI-generated code.