Peterbe.com

My Django skills are getting rusty (because I haven't used it for any work projects in years) but I heard you can scrap PgBouncer and django-db-connection-pool and just set:

  ...
  "CONN_MAX_AGE": 0,
  "OPTIONS": {
      "pool": True,
  }
  ...

...on the settings.DATABASES and that's all you have to do.

Simple benchmark

I created a very simple view that does a very simple Django ORM query. It looks like this:

def db_debug(request):
    query = Category.objects.all().order_by("-id")
    for x in query.values_list("id", flat=True)[:1]:
        return json_response({"last_id": x})

Then, I ran a benchmark against http://localhost:8000/api/v1/__db_debug__ and record the requests per second.

With NO connection pooling

I.e.

{'CONN_HEALTH_CHECKS': False,
 'CONN_MAX_AGE': 0,
 'DISABLE_SERVER_SIDE_CURSORS': False,
 'ENGINE': 'django.db.backends.postgresql',
 'HOST': 'localhost',
 'NAME': 'peterbecom',
 'PASSWORD': '',
 'PORT': '',
 'USER': ''}

Using oha, running it a bunch of times, this is what I get:

❯ oha -n 1000 http://localhost:8000/api/v1/__db_debug__
Summary:
  Success rate: 100.00%
  Total:    5134.5693 ms
  Slowest:  342.7820 ms
  Fastest:  13.7949 ms
  Average:  250.5163 ms
  Requests/sec: 194.7583

Then, I change the setting to:

{'CONN_HEALTH_CHECKS': False,
 'CONN_MAX_AGE': 0,
 'DISABLE_SERVER_SIDE_CURSORS': False,
 'ENGINE': 'django.db.backends.postgresql',
 'HOST': 'localhost',
 'NAME': 'peterbecom',
+'OPTIONS': {'pool': True},
 'PASSWORD': '',
 'PORT': '',
 'USER': ''}

Same oha run:

❯ oha -n 1000 http://localhost:8000/api/v1/__db_debug__
Summary:
  Success rate: 100.00%
  Total:    948.3304 ms
  Slowest:  58.2842 ms
  Fastest:  4.0791 ms
  Average:  46.2348 ms
  Requests/sec: 1054.4848

In both benchmarks, I use gunicorn with wsgi and 2 workers.

Side note

As a "baseline check", what would the Django view (using WORKERS=2 gunicorn), yield if you don't do any database queries at all? I changed the benchmark Django view to:

def db_debug(request):
    return json_response({"max_id": 100})

and run the benchmark a bunch of times:

❯ oha -n 1000 http://localhost:8000/api/v1/__db_debug__
Summary:
  Success rate: 100.00%
  Total:    630.1889 ms
  Slowest:  37.6904 ms
  Fastest:  2.7664 ms
  Average:  30.7311 ms
  Requests/sec: 1586.8259

Conclusion

Adding 'OPTIONS': {'pool': True}, to the DATABASES['default'] config made this endpoint 5.4 times faster.

Doing that one simple SQL query makes that view 1.5 times slower, which makes sense because it's doing something.

Questions

I don't know why but I had to switch to psycopg[binary,pool]>=3.2.9 to make this work. Before, I used to use psycopg2-binary==2.9.10.

To be honest, I don't know why setting...

DATABASES["default"]["MAX_CONN_AGE"] = 60

...instead of the default 0, didn't make it better.

I made a web app that helps you extract screenshots from a video file. You technically don't "upload" it but you select a video file from your computer into the web app, and the screenshots are generated more or less instantly.

Canvas Web API

Why did I make this app? Because I wanted to experiment with the Canvas (Web) API and how it can be combined with a video element. I originally typed into some AI prompt and got most of the code from that, but I felt like I didn't understand it. And to be able to quickly iterate and play with it, I ended up making a simple web app so that I can tune it.

How it works

The gist of the code is this:

export function createVideoThumbnail(
  videoFile: File,
  options: Options,
): Promise<string> {
  const { quality = 1.0, captureTime = 0.1, format = "image/jpeg" } = options
  return new Promise((resolve, reject) => {
    const video = document.createElement("video")
    const canvas = document.createElement("canvas")
    const ctx = canvas.getContext("2d")

    if (!ctx) {
      reject(new Error("Failed to get canvas context"))
      return
    }

    video.preload = "metadata"
    video.muted = true
    video.playsInline = true

    const videoUrl = URL.createObjectURL(videoFile)
    video.src = videoUrl

    video.onloadedmetadata = () => {
      const width = video.videoWidth
      const height = video.videoHeight
      canvas.width = width
      canvas.height = height
      video.currentTime = captureTime
    }

    video.onseeked = () => {
      ctx.drawImage(video, 0, 0, canvas.width, canvas.height)
      const dataURI = canvas.toDataURL(format, quality)
      URL.revokeObjectURL(videoUrl)
      resolve(dataURI)
    }

    video.onerror = () => {
      URL.revokeObjectURL(videoUrl)
      reject(new Error("Error loading video file"))
    }

    video.load()
  })
}

See src/create-video-thumbnail.ts on GitHub.

The project

The code for it is here: https://github.com/peterbe/video-to-screenshots

The web app is hosted on Firebase Hosting. The foundation of the code is React with react-router, and uses pico css. I use Bun and Vite to run and build the app.

Conclusion?

It works and it's basic. You can download the JPEGs. It's not very pretty. It's a weekend project and it accomplishes something.

But it doesn't work in Safari. Anybody got any ideas?

Perhaps it would be cool to allow the user to see many many more thumbnails and allow them to specify more exactly which capture times to make screenshots out of. What do you think?

This can come in handy if you're working with large Python objects and you want to be certain that you're either unit testing everything you retrieve or certain that all the data you draw from a database is actually used in a report.
For example, you might have a SELECT fieldX, fieldY, fieldZ FROM ... SQL query, but in the report you only use fieldX, fieldY in your CSV export.

class TrackingDict(dict):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._accessed_keys = set()

    def __getitem__(self, key):
        self._accessed_keys.add(key)
        return super().__getitem__(key)

    @property
    def accessed_keys(self):
        return self._accessed_keys

    @property
    def never_accessed_keys(self):
        return set(self.keys()) - self._accessed_keys

Example use case:

user = {
    "name": "John Doe",
    "age": 30,
    "email": "jd@example.com",
}
user = TrackingDict(user)
assert user["name"] == "John Doe"

print("Accessed keys:", user.accessed_keys)
print("Never accessed keys:", user.never_accessed_keys)

This will print

Accessed keys: {'name'}
Never accessed keys: {'email', 'age'}

This can be useful if you have, for example, a pytest test that checks all the values of a dict object and you want to be sure that you're testing everything. For example:

assert not user.never_accessed_keys, f"You never checked {user.never_accessed_keys}"

UPDATE (June 16)

Here's a typed version, by commenter Michael Cook:

from typing import TypeVar, Any

K = TypeVar('K')
V = TypeVar('V')

class TrackingDict(dict[K, V]):
    """
    A `dict` that keeps track of which keys are accessed.
    """

    def __init__(self, *args: Any, **kwargs: Any) -> None:
        super().__init__(*args, **kwargs)
        self._accessed_keys: set[K] = set()

    def __getitem__(self, key: K) -> V:
        self._accessed_keys.add(key)
        return super().__getitem__(key)

    @property
    def accessed_keys(self) -> set[K]:
        return self._accessed_keys

    @property
    def never_accessed_keys(self) -> set[K]:
        return set(self.keys()) - self._accessed_keys

Reminder;
SSG - Static Site Generation
SPA - Single Page App

I have a lovingly simple web app that is an SPA. It's built with Vite, React, and React Router. In essence, it looks like this:

// src/main.tsx

import { createRoot } from "react-dom/client";
import { BrowserRouter } from "react-router";
import AppRoutes from "./AppRoutes";

const root = document.getElementById("root");

createRoot(root).render(
  <BrowserRouter>
    <AppRoutes />
  </BrowserRouter>
);

(full code GitHub)

and:

// src/AppRoutes.tsx

import { Route, Routes } from "react-router";
import { About } from "./About";
import { App } from "./App";
import { Layout } from "./Layout";

export default function AppRoutes() {
  return (
    <Routes>
      <Route element={<Layout />}>
        <Route path="/" element={<App />} />
        <Route path="/about" element={<About />} />
      </Route>
    </Routes>
  );
}

(full code on GitHub)

Last week I started a side project called PissueTracker which is a web app that lets you create issues/bugs. What's cool about it is that it uses an offline database, namely IndexedDB in the browser. That means that every time you create or edit an issue, it's stored in a database in your browser. The problem with this is that inevitably, you might reset your entire browser. Or, when auth is working, you might want to access your data from an entirely different browser. So I'm adding a client-to-server sync. (It'll become two-way sync later)

How it works is demonstrated with this code:

db.issues
  .update(issue.id, { state: newState, syncedAt: null })
  .then((updated) => {
    if (updated) {
      notifications.show(...);
      mutate(issue.id);

The payload sent to the .update() is first the ID, and then it's an object. No matter what the payload changes are, state: newState in this case, there's always syncedAt: null tacked on.
Then, after it has successfully updated the local database, it triggers that mutate function. Let's dig into that next.

The mutate function is based on the useMutation hook in TanStack Query which is a wrapper for making a fetch XHR POST request. It looks like this:

const issue = await db.issues.get({ id });
const response = await upfetch(`/api/sync/issue/${tracker.uuid}`, {
  method: "POST",
  body: { id, issue },
});
if (response.ok) {
  db.issues.update(id, { syncedAt: new Date() });
  return response.json();
}
throw new Error("Failed to sync issue");

Get it?
First, we set syncedAt: null into the local browser database.
Then, we send the issue object to the server. Once that's done, we update the browser database, again, but this time set syncedAt to a valid Date.

Once all of this is done, you have a copy of the same issue payload in the server and in the client. The only thing that is different is that the syncedAt is null on the server.

The server is an Hono server that uses PostgreSQL. All records are stored as JSONB in PostgreSQL. That means we don't need to update the columns in PostgreSQL if the columns change in the Dexie.js definition.

UPDATE

This idea is flawed. The problem is the use of incrementing integer IDs.
Suppose you have 2 distinct clients. I.e. two different people. One on a train (in a tunnel) and one on an airplane. If they both create a new row, they'd be sending:

id: 2
title: "Trains are noisy"
body: ...

and

id: 2
title: "Airplanes are loud"
body: ...

That would override each other's data when synchronized with the central server, later.

The only solution is UUIDs.

I've started a new side project called PissueTracker. It's an issue tracker, but adding the "P" because my name is "Peter" and it makes the name a bit more unique. Also, the name isn't important. What this project can do is important.

In a sense, it's a competitor to GitHub Issues. An issue can be anything such as a feature or bug report or just something that needs to be addressed. In another sense, it's a competitor to a web product I built 24 years ago called IssueTrackerProduct. That was a huge success and it was so much fun to build.

What's special about this issue tracker is that it's all about offline data, with server sync. All the issues you create and edit are stored in your browser. And if you don't want to lose all your data if/when your browser resets, you can have all that data constantly synchronized to the server which will either store it in PostgreSQL or the file system of a server. Since there's a shared data store, you can also authorize yourself amongst others in the system and this way it can become a collaborative experience.

I'm building it by using a React framework called Mantine, which I'm a huge fan of. The browser storage is done with Dexie.js which uses IndexedDB and allows you to store data in relational ways.

So far, I've only spent 1 weekend day on it but I now have a foundation. It doesn't support comments or auth, yet. And I haven't even started on the server sync. But it supports creating multiple trackers, creating/editing issues, fast filtering, and safe Markdown rendering.

The front end is handled by Vite, React Router, and Bun, I don't know if that's important. Right now, the most important thing is to have fun and build something that feels intuitive and so fast that it feels instantly responsive.

More to come!

List of issues in a tracker

Viewing an issue

UPDATE (Mar 20, 2025)

Blogged about the client-to-server sync here: "Adding client-to-server sync to PissueTracker."

Spot the Difference is a web app where you're shown two snippets of code (programming, config files, style sheets, etc) and you're supposed to find the one difference. If you get it right, you get showered in confetti.

This started as an excuse to try out some relevant patterns in React that I was working on. I was also curious about writing a SPA (Single Page App) using Bun and React Router v7 (as a library). It started as a quick prototype, with no intention to keep it. But then I found it was quite fun to play and I liked the foundation of the prototype code; so I decided to clean it up and keep it. After all, it's free.

There's a goose on the home page because when I first showed my son the prototype, I said: "It's a silly goose game. Wanna try it?". So it stuck.

The technology behind it

Almost embarrassing that I didn't realize that it's the same thing!

class MyClass {
  get something() {
  }
}

is the same as Python's property decorator:

class MyClass
    @property
    def something:

They both make an attribute on a class automatically "callable".
These two are doing the same functionality:

class Foo {
  get greeting() {
    return "hello";
  }
  place() {
    return "world";
  }
}

const f = new Foo();
console.log(f.greeting, f.place());
// prints 'hello world'

and

class Foo:
    @property
    def greeting(self):
        return "hello"

    def place(self):
        return "world"

f = Foo()
print(f.greeting, f.place())
# prints 'hello word'

tl;dr; Use data() in your loader function and make your headers function pass it on to get headers be dependent on what's happening in the loader function.

I recently rewrote the front end of this website from Remix to react-router v7. A route is a page, which can be something like /about or have a parameter in it like /blog/:slug.

The way react-router v7 (the "framework mode") works is that your route looks like this:

import type { Route } from "./+types/post"

export async function loader({ params }: Route.LoaderArgs) {
  const post = await fetchPost(params.slug)
  return { post }
}

export default function Component({loaderData}: Route.ComponentProps) {
  return <h1>{loaderData.post.title}</h1>
}

So good for so far. But suppose you want this page to have a certain header, depending on the value of the post object. To set headers, you have to add an exported function called, surprise surprise; headers. For example: