Using React with canvas, WebGL and custom renderers.

I love React for making UI and I love HTML5 canvas as a general drawing API. Sadly they don't play well together if you follow the naïve route.

Because of React's functional nature and functional paradigm causing so many re-renders it can get a little awkward working with the canvas API and associated libraries. Here's a good approach I don't use, and a detailed explanation of what I do use.

If you learn by reading code, I have a small Open Source project called Isodraw that uses everything I speak about here. It can be found here.

One approach I don't use

React Three Fiber (R3F) takes away a lot of this pain by managing canvas state and hooking React component lifetimes up to the adding and removing of objects from a scene. It's generally really awesome. For smaller projects where I need less control, I quickly reach for it and its excellent ecosystem. However, I'm not a fan for larger projects.

Although R3F is perfectly capable of doing large and complex projects, I find I don't gel with that way of thinking, and it forces me into patterns I don't enjoy. I dislike having scene logic mixed with component state. I like having more control of data layout and a good idea of what is happening when. I also found myself using a mixture of R3F's 'happy path' and its escape hatches quite often (especially when dynamically loading models), making what felt like an inconsistent and somewhat messy codebase. My final problem is that I don't want to be so tightly integrated with React; coupling the UI layer with the 'core engine' removes future flexibility if I ever want to swap UI framework.

What I do instead

As as example app, I am going to use a drawing tool. This tool has the ability to draw lines on an HTML canvas as well as pick colors, fill and delete shapes.
I will provide the source code from a real SVG drawing app I have made with this article.

I prefer the following setup:

A React front end, a canvas engine (e.g ThreeJs, PixiJs, Pex-renderer or something hand written) and between them a shared store where I put all application state. I make sure React re-renders when relevant state changes, and there are callbacks to the canvas engine when relevant state changes. It takes some co-ordination but this is how I tackle it.

Initialising the engine in React

First step is getting our engine and React to talk to each other. We need to initialise our CanvasEngine class, but we have to do it carefully. If we just created it in the component body (let engine = new CanvasEngine()), it would be re-instantiated on every single re-render. The solution is to use a combination of useRef and useEffect.

First, we set up two refs. One for the <canvas> DOM element itself, and another to hold our CanvasEngine instance.

const canvasRef = useRef(null);
const engineRef = useRef(null);

useRef gives us a "box" that persists for the entire lifetime of the component. Anything we put in ref.current will still be there after a re-render and updating it won't trigger a new render. This makes it perfect for holding onto things like DOM nodes and class instances. You can find useRef Docs here.

Next, we use useEffect with an empty dependency array [] telling React to run this code only once, right after the component has mounted to the page."

Inside the effect, we perform a simple check: is the canvas element available, and have we not yet created our engine? If both are true, it's safe to create a new CanvasEngine instance and store it in our engineRef.

function App() {
  const canvasRef = useRef<HTMLCanvasElement>(null);
  const engineRef = useRef<CanvasEngine | null>(null); 

 useEffect(() => {
  // If the canvas element exists and our engine ref is empty...
  if (canvasRef.current && !engineRef.current) {
    // ...create the engine and store the instance in the ref.
    engineRef.current = new CanvasEngine(canvasRef.current);
  }

  // We can also return a cleanup function from useEffect.
  // This will run when the component unmounts.
  return () => {
    engineRef.current?.destroy(); // A good place to clean up listeners.
  };
}, []); // The empty array ensures this effect runs only once.

  return (
    <main className="app-container">
        <canvas ref={canvasRef} ></canvas>
    </main>
  );
}

With this setup, we have a stable, persistent reference to our engine instance that we can access anywhere in our component, all while guaranteeing it's only created a single time.

How React components interact with the store

I create a store with Zustand, here we store all application state as well as the actions to make modifications. In react we can use the hooks Zustand provides to be reactive and re-render when changes occur.

First lets create our store using Zustand, calling it useDrawingStore

export const useDrawingStore = create(subscribeWithSelector((set, get) => ({
  ... store 
})

And now use it in a React component, for example this color palette:

import React from 'react';
import { useDrawingStore } from '../store/drawingStore';
import { shallow } from 'zustand/shallow';

const ColorPalette = () => {
  const { paletteColors, currentColor, setCurrentColor } = useDrawingStore(state => ({
    paletteColors: state.paletteColors,
    currentColor: state.currentColor,
    setCurrentColor: state.setCurrentColor,
  }), shallow);

  const handleColorChange = (e) => {
    setCurrentColor(e.target.value);
  };

  return (
    <section >
        <input
          type="color"
          value={currentColor}
          onChange={handleColorChange}
          title="Select Color"
        />
    
      {paletteColors.map((color, index) => (
        <div
          key={`${color}-${index}`}
          style={{ backgroundColor: color }}
          title={`Select color ${color}`}
          onClick={() => setCurrentColor(color)}
        />
      ))}
    );
};

export default ColorPalette;

You can see in the element code we subscribe to the store and get the colors associated with the palette. If you click on that color it sets the value in the store and the element re-renders. If that color is changed in the store by something else (in our case the canvas engine) then the element also re-renders. So our React component re-renders correctly regardless of where the state change happens.
Notice the shallow as the useDrawingStore argument, this allows the component to do a shallow comparison of the values retrieved to see if it needs to update component state. Simply put, it means the component will only re-render when something relevant changes. Without this, it would re-render when anything in the store changes. This is because our selector creates a new object every time it runs. If we were only selecting a single value, like useDrawingStore(state => state.activeTool), we wouldn't need shallow as the default comparison works fine for simple values like strings."

How the engine interacts with the store

So, we've established how the React part of the system updates in response to store changes. How does our canvas engine interact, lets assume we have this action in the store:

export const useDrawingStore = create(subscribeWithSelector((set, get) => ({
  // ... all our state definition
  // all other actions 
  cancelDrawing: () => {
    set({ currentDrawing: { isDrawing: false, points: [] } });
  },
  // etc

  })

we can call it directly from our engine:

// you can just import the store, no need for DI
import { useDrawingStore } from '../store/drawingStore';


...
  handleKeyDown(e) {
    useDrawingStore.getState().cancelDrawing();
  }
...

and we can use the subscribe method on the store within the engine to call that classes methods, like so:

  useDrawingStore.subscribe(
      // Select relevant state slices
      (state) => [state.shapes, state.currentDrawing, state.activeTool,
                  state.selection, state.snapPoint],

      // Callback to redraw
      () => this.redrawAll(),

      // Options, 
      // equality function tells when to trigger the callback
      { equalityFn: (a, b) => JSON.stringify(a) === JSON.stringify(b) }
    );

Quite neat really.

No matter what triggers a change in data in our store, any relevant engine methods will be run and any relevant react components will re-render.

Accessing the engine functions from a React component

I sometimes need to trigger things in the engine without interacting with the store. An example I have come across is importing a file and processing it for the in the app. In Isodraw (linked below) you can import SVG's that have been made in Isodraw back into the software for editing. When this happens the upload element directly passes the SVG to the engine which processes it to the current data structure and passes that to the store. I can pass the ref of the engine to an element and trigger the method from there.

ts// in our app declaration
function App() {
  ...
  const engineRef = useRef<CanvasEngine | null>(null);

  ...
  return (
      <Toolbar engineRef={engineRef} />
  );
}

// and in the toolbar element

interface ToolbarProps {
  engineRef: React.RefObject<CanvasEngine | null>;
}

const Toolbar = ({ engineRef }: ToolbarProps) => {
  ....
  const handleImportClick = () => {
    if (fileInputRef.current) {
      fileInputRef.current.click();
    }
  };

...
})
        

This allows us to directly call our engine methods from an element.

This is the structure used by my open source isometric SVG drawing tool, Isodraw, which you can find here. It's also the same structure that has been rolled out for an avatar system in my current job building SwiftKitchen, that is used by around 15,000 parents,

If you have any questions then my DM's are open @simonharrisco