1.5. Font

ConsoleUser does not attempt to “read” the screen, that is, it does not try to make sense of things on the screen. ConsoleUser only looks for elements it expects to find and tries to match them. This is straightforward when the elements are typical visual objects like buttons or icons. Text can be more difficult.

One aspect that makes text more difficult to match is that we don’t always know in advance what the text will be. For example, sometimes we may want ConsoleUser to click on an email with a given subject, but that subject may not be known until runtime. In that case, we will need to be able to take a string and convert that to the pixels that will display them on the screen.

That process has its own challenge, especially since different applications and different platforms use different fonts to render the text. That is, the same string may look very different depending on what is drawing it on the screen. Often we will know what font will be used at the time we want to match it, so ConsoleUser preserves a collection of fonts that it can use to internally build what the rendered region should look like.

Any collection like this is going to be incomplete–especially since the size and weight of the font can change how it looks drastically. So sometimes we have to add a new font to the collection. This can be a tedious process, but is not hard once some initial bits are figured out.

Steps:

  1. Identify the font/weight/size (E.g., is it Segoe UI Bold 12pt?)

  2. Display a “clean” version of that

  3. Capture that screen

  4. Figure bounds of characters

  5. Cut the characters out to their own files

  6. Add to database

1.6. Identifying Font

Unfortunately, there is not a single universal way to do this. Sometimes it can be found in documentation or online (like what the default font for an OS is often). Sometimes it can be set in the application to something predictable. Sometimes an inspector, like in a browser, can be used to see what it is.

Ultiamtely, one way or another the font must be identified, including its size and wether it is bold or not (or in between if the font supports it).

1.7. Display That Font

The rest of the steps require having clean images of each character in the font. This can be challenging when the target application renders them antialiased, as that blurs all the edges and makes that unsuitable for our needs.

The most common way to do this is to create a web page that contains all of the characters with a predictable way to tell the bounds of each glyph. Since web pages can specify their font, this page can display the characters in the desired font. The default SyntheticInternet includes a file like this. In www.zulu.isp’s directory, the current default config puts that in /var/www/html-https_50/disk2/www.zulu.isp, contains alpha.html. That has all of the characters separated by a pipe character.