Tag: image classification

I needed higher resolution screenshots for an ML model to classify elements on a webpage, but the default Lighthouse screenshot was too compressed. So, I created a custom Lighthouse Gatherer using Puppeteer. This gatherer captures a full-page, high-resolution screenshot encoded as base64 and returns it along with the device pixel ratio. This was a fun little project, and the code is surprisingly concise. However, future Lighthouse versions may include higher-resolution screenshots, making this gatherer redundant.

I built a web app using Deno, Fresh, and TensorFlowJS to classify images as links or buttons. The app uses a pre-trained ML model and allows users to drag and drop multiple images for classification. I encountered challenges with server-side rendering and islands, specifically with integrating a file-drop web component. I also documented the process of integrating the TensorFlowJS model, including model loading and prediction handling. The code is available on GitHub.

In this project, I'm working on an accessibility tool to detect links styled as buttons, a common issue that can confuse users. My approach involves scraping websites to gather images of buttons and links, and then training a machine learning model to distinguish between them. This post focuses on the scraping process using Puppeteer. I encountered challenges like occluded elements and smooth scrolling, which I addressed by checking for occlusion and disabling smooth scrolling. The next step is training the ML image classifier.