read_lcd_from_pictures-WEB

Do Joulies Work?

Reading an image of a LCD display with Image Classification in Mathematica

Justin Pearson

2014-10-26

Summary

Joulies are a product meant to maintain hot coffee at a comfortable temperature longer. They are sealed metal sphereoids that contain a proprietary material that melts at 140°F. The heat of fusion helps maintain the coffee’s temperature. You keep them in your coffee cup and dump your coffee over them. They displace about 1.5 Tbsp per Joulie.

We took two coffee cups, one with 5 Joulies and one without, and filled them with boiling water to roughly the same level. We measured their temperatures with two digital thermometers. We took pictures of the thermometers with two different cameras: Dad’s Leica handheld digital camera was set to “time lapse” and snapped a picture every minute, and my Macbook Pro took a picture about every 10 seconds.

We also had a Raspberry Pi with temperature sensors. However, water leaked into the sensors and corrupted their measurements.

Afterward, Dad manually recorded the temperature of the two thermometers from each of this 30 images.

The Macbook took 180 images. Instead of doing the obvious thing of just looking at the pictures and writing down the temperatures, I used this as an opportunity to test Mathematica’s ability to classify the LCD digits.

Specifically, I wanted to see if Mathematica’s Classify() function could identify the digits on the two LCD thermometers. I found that the classification works well for a single LCD segment (eg the one’s digit) but a classifier trained on the one’s digit doesn’t necessarily do well on the ten’s or hundred’s digits, and a classifier trained for one thermometer doesn’t work on another.

I haven’t answered the main question -- do Joulies work -- but it was fun playing around with various Mathematica image-processing algorithms.

Import pictures

Ugh, MacBook flips it so the camera looks like a mirror. Correct:

Pic description:
img 48.tiff - both reading room temp - 67deg F
img 54.tiff - begin to pour into ‘joulie’ cup
57 - begin to pour into ‘control’ cup
58 - fog on lens
227 - last pic before ‘control’ therm turns off

Get Digits of ‘Joulie’ cup’s thermometer

Trim

Crop (trim) the image to this bounding box around the Joulie therm LCD:

Now we have a trimJoulie() function:

Fix Perspective

Undo the skew.

Will a simple rotation work?

The digits seem unskewed according to the gridlines, but the edges of the screen still seem skewed.

Manual unskewing with a Perspective Transformation

Have Mathematica find a geometric transform to fix the perspective

Now we have a fixPerspectiveJoulie() function:

Extract digits from picture

Bounding boxes around three LCD digits:

Now we have a getDigitsJoulie() function:

Compute all digits

(Note: @* is ‘function compose’)

Classify digits as integers

First make a training set.

Put it on auto-play, watch it and type the numbers as they play (I put my fingers on 1,2,3,4 and 7,8,9,0). Starting at index 15, what’s the least-sig digit?

Note: Coulda just repeated this for the whole file. But if it was really long, couldn’t.

Manually add some mis-classified points

Build classifier

How does it work? We trained it mostly on the one’s place LCD segment. Perhaps it will work on the ten’s and hundred’s segments too?

One’s place:

Ten’s place:

Hundred’s place:

Tab view:

Awesome.

Not bad.

Get Digits of Control thermometer

Repeat the above tweaking for the ‘Control’ thermometer.

Trim

Crop (trim) the image to this bounding box around the Control therm LCD:

Extract digits from picture

Bounding boxes around three LCD digits:

Compute all digits

Classify digits as integers

Put it on auto-play, watch it and write it down. Starting at index 13, what’s the least-sig digit?

Note: Coulda just repeated this for the whole file. But if it was really long, couldn’t.

Build classifier

Assess classifier

How does it work? We trained it mostly on the one’s place LCD segment. Perhaps it will work on the ten’s and hundred’s segments too?

One’s digit:

Seems pretty good!

Ten’s digit:

Not great.

Hundred’s digit:

Since it gets so many of the tens’ and hundred’s wrong, let’s input them manually.

Manually find the hundreds’ and tens’ digits

The above classification is very sensitive to brightness of the images. The classifier was trained from the one’s digit, and sometimes it doesn’t do very well on the ten’s or hundred’s digits. Since the hundred’s and ten’s place doesn’t change very quickly, it’s not much trouble to just write them down manually.

The format x,y→z means: frames y thru z display digit x.

Ten’s place:

Hundred’s place:

Future work: Another workaround: use the one’s digit 0->9 transition to tell when the ten’s digit should change

The one’s digit is well-classified after it stops bonking around after index 15 or so. So let’s just enter manually for 1-15 and then rely on the one’s digit to tell us when it’s changing:

Plot results

Import Dad’s data

I shifted dad’s time axes to get it to line up:

Plot all data

Future work: Calculate efficiency

JPP: I think we still need to know how much water was in each of the cups. . . . Dad can measure the volume of his mugs.

Dad: Up to the water level visible in the Joulie-filled cup is 276 ml, give or take about 5%.

Spence: The Joulie information packet says: “0.75oz/1.5Tbsp displaced per Joulie; 4oz/118mL regulated per Joulie.”

Specific heat of water:

TODO: more on this.

Created with the Wolfram Language