Last week I got a trained a simple model that will detect if an image looks like a link or a button. It was fun and I learnt a lot.

This article is less about the ML model and more about how I made a TensorflowJS Preact Component using Deno and Fresh and some of the snags I hit so that you don't have to hit them.

The Demo itself might not make sense, so I will give some context first. I trained the ML model in Google Colab, and while it has a handy little script that lets me upload files it is incredibly slow, so I wanted a way to drag and drop many images on a page and have it automatically classify the images. If you try the demo just take a screen shot of a button or a link with about 10px of padding around it like so ) and it will output something like the following:

I also wanted an excuse to get a model into "production".

The Code is built around Deno and Fresh, and broadly it was a lot of fun to get started. I really liked how quick it is to deploy (feels significantly faster than Vercel). I did struggle with the documentation, for example the docs around islands are sparse and it implies that it runs in the client - so I naturally thought I could just include my file-drop Custom Element, however I ran in to a number of problems where because it is rendering on the server it doesn't have access to DOM Element APIs and thus would not buidl. I had to rely on a handy hint from Luca Casonato.

if (IS_BROWSER) {
  await import("https://esm.sh/file-drop-element");
}

After that, using the Web Component in an island is just like using it in the browser.

interface FileDropProps {
  accept: string;
  multiple?: boolean;
}

export default function FileDrop(props: FileDropProps) {
  const [accept] = useState(props.accept);
  const [multiple] = useState(props.multiple);
  const tensorFlow = useRef(null);
  let [files, setFiles] = useState("");
  let x: FileDropEvent;

  const onFileDrop: FileDropEvent = (event) => {
    setFiles(event.files);
  }

  const onFileSelect: Event = (event) => {
    setFiles(event.target.files);
  }

  return (
    <div>
      <file-drop onfiledrop={onFileDrop} accept={accept} multiple={multiple}>
	...
  )
}

Although now that I think about it, is it just a functionless element sent on the request? I should check... It feels like the docs could be a little clearer.

Integrating TensorFlow was surprisingly straight forwards (and can be seen completely here).

The first thing that I had to do was include Tensorflow as a global <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@latest/dist/tf.min.js"></script> on my index route, and then I created the component as follows.

export function TensorFlow({ url, files }) {
  const [model, setModel] = useState();
  const [predictions, setPredictions] = useState();

  useEffect(async () => {
    const newModel = await tf.loadLayersModel(url);

    setModel(newModel);
  }, [url]); // When the URL changes, refetch the model

  useEffect(async () => {
    console.log("Analysing", files);

    const predictionsResults: PredictionResult[] = []
    for (const file of files) {
      const fileUrl = URL.createObjectURL(file)
      predictionsResults.push({ prediction: await analyse(fileUrl, model), fileUrl });
    }

    setPredictions(predictionsResults);
  }, [files]);

  return (
    <div>
      <h2>Prediction </h2>
      <div>{model == null ? 'Loading Model' : (predictions != null) ? renderPredictions(predictions) : ''}</div>
    </div>
  );
}

And it can be used as follows <TensorFlow url="/model/model.json" files={files} ref={tensorFlow}></TensorFlow>.

I was quite happy with this structure because it allows you to control which model TensorFlow should use, and which files it accepts (which is specific to my usecase).

The only thing that is special here is that I only want to instantiate the model when the property changes, and I only want to run the prediction when the input files array changes.

And that's it. It just worked.

Next up will be integrating this Tensorflow model into a custom Lighthouse Audit.

I lead the Chrome Developer Relations team at Google.

We want people to have the best experience possible on the web without having to install a native app or produce content in a walled garden.

Our team tries to make it easier for developers to build on the web by supporting every Chrome release, creating great content to support developers on web.dev, contributing to MDN, helping to improve browser compatibility, and some of the best developer tools like Lighthouse, Workbox, Squoosh to name just a few.

I love to learn about what you are building, and how I can help with Chrome or Web development in general, so if you want to chat with me directly, please feel free to book a consultation.

I'm trialing a newsletter, you can subscribe below (thank you!)

Enter your email

Mastodon Book a consultation