ericek111/myscan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Global page assembly with loop closures and lens undistortion

Browse Source 
The frame-by-frame stitcher accumulated drift, so later passes landed rotated
and offset, doubling text. myscan/pagegraph.py instead solves all frame poses
together: ECC odometry between consecutive frames (bridging single corrupt
frames), SIFT+RANSAC loop closures between overlapping passes, and a robust
Gauss-Newton pose graph. Loop matches must agree over the whole overlap, which
rejects matches on repeated text. Off-page frames and frames that disagree
with the mosaic are left out of the render.

calib.Undistort applies the EEPROM lens model (OpenCV k1..k3, p1, p2 plus
the tilt homography, as decoded in EEPROM.md); up to 8.8 px at the corners.

scan.py registers frames on a worker thread so capture no longer drops
frames, then assembles the page after the stop press (~11 s for a full
page). stitch.py uses the new assembly by default; --live keeps the old one.

EEPROM.md documents the full calibration blob layout, checksum and signature.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
erik
2026-10-02 13:06:45 +00:00
co-authored by Claude Opus 5.5
parent e70d56d957
commit 026aa6c894
8 changed files with 781 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
EEPROM.md
+298
View File
@@ -0,0 +1,298 @@
# EEPROM contents (Hama myScan / Dacuda LuckyCat)
This is a field-by-field map of the per-device calibration blob, from one unit's dump (`calib/eeprom_3000.bin`).
For the read protocol, see `FINDINGS.md` ("Per-device calibration in EEPROM"). For the app side, see
`RE_colour_stitch.md` section 1.4.
Status tags:
- **[P]**: parsed by `myscan/calib.py`.
- **[D]**: decoded by hand from the bytes; the value is clear.
- **[G]**: guessed field name or meaning.
- **[?]**: unknown.
- **[V]**: verified in disassembly or decompilation; the function address is given as evidence.
## EEPROM map
| Address | Size | Content |
|---|---|---|
| 0x3000 | 2 | u16 LE blob length N (19417 on this unit). Valid if 0 < N <= 20000 (0x4E20) [V 0x138c4e0, read of 0x3000 at 0x138c743, bound check 0x138c7a0]. The write path rejects N >= 0x4FB7 [V 0x138cc90] |
| 0x3002 | 4 | u32 LE standard CRC-32 (zlib polynomial) over the N blob bytes, signature included. Verified in Python against the dump: 0x3cd95f42 [D]. The app **does not recompute or compare it on read**: it is only a cache key (see below) [V 0x1335430] |
| 0x3006 | N | `boost::archive::binary_iarchive` blob (i386 sizes), then a 256-byte RSA signature at its tail |
| 0x7EB5 | 64 | mouse ID: raw 64-byte buffer, handled as a string. Read by `OVMouseManager::readMouseId` (0x138e1e0 [V], read at 0x138e4fa), written by `writeMouseId(const char*)` (0x138db20 [V], write at 0x138dd80). **Not yet read from the device.** The `HalLuckyCat` wrapper is 0x1334a60 |
| < 0x3000 | ? | **The app never touches anything below 0x3000.** A scan of every call to `OVIF_DaEEPROMRead/Write` (the stubs 0x17ff6fa / 0x17ff700) in the main binary gives only the callers 0x138c4e0 (0x3000, 0x3006), 0x138d410 (0x3002), 0x138e1e0 (0x7EB5), 0x138cc90 (`writeBinaryData`: a single write starting at 0x3000 for header + checksum + blob, call at 0x138cf3f; the caller builds the whole buffer, the CRC is not computed there) and 0x138db20 (0x7EB5). UVCControl.framework just wraps the raw read/write (`EEPROMRead:withStartAddr:withLength:`). So whatever is below 0x3000 is firmware-side only [V by absence] |
Everything the app reads from the EEPROM is therefore: length, checksum, blob and the 64-byte ID.
## Blob layout
All offsets below are relative to the blob start (EEPROM 0x3006). The whole blob parses with **no gaps**
(a stand-alone parser consumes exactly bytes 0 .. N-256, then the signature follows).
Stream = `boost::archive::binary_iarchive`, library version 7 (boost ~1.42), i386 sizes. Each class is written
as: `tracking u8`, `class version u8`, then its members, in the order of `serialize<binary_iarchive>`.
All five classes here have `00 0e` = tracking 0, **class version 14**. Fixed C arrays (`T[n]`) are written as
`u32 count` + elements. `std::vector<T>` is `u32 count`, `u32 item_version`, elements. A class that is
first met inside a container (`std::vector`, `boost::array`) writes its own `00 00` (tracking 0, version 0) once.
The app accepts class versions 7..15 only (else `WrongArchiveVersionException`, "Incompatible archive version!",
[V] 0x14cc1a0 and the other four `serialize` functions).
Serialize functions ([V], `serialize<binary_iarchive>`): Info 0x14cc1a0, Color 0x14cde20, Undistortion 0x14cefe0,
Sensor 0x14cfcd0, Roi 0x14d0990. Names come from the property-tree writers: Undistortion 0x14c0510, Sensor 0x14c1300,
Roi 0x14c23a0, Info 0x14c2bc0, Color 0x14bedb0 (all [V]); the umbrella `HardwareData` has members
color @0x00, undistortion @0x68, sensor @0xF4, roi @0x144, info @0x158 (0x14be690 [V]).
### Header
| Offset | Value | Meaning |
|---|---|---|
| 0x00 | u32 22, `serialization::archive` | boost signature [D] |
| 0x1A | `07 04 04 04 08` | library version 7; sizeof int/long/float/double = 4/4/4/8 [D] |
| 0x1F | u32 1 | endianness check [D] |
### RoiParams (0x23: `00 0e`) [V 0x14d0990, names 0x14c23a0]
Five plain i32 in this order, no version conditions.
| Offset | Member offset | Type | Name | Value |
|---|---|---|---|---|
| 0x25 | +0x00 | i32 | `yBaseOffset` | 105 |
| 0x29 | +0x04 | i32 | `roiX0` | 0 |
| 0x2D | +0x08 | i32 | `roiY0` | 0 |
| 0x31 | +0x0C | i32 | `roiX1` | 640 |
| 0x35 | +0x10 | i32 | `roiY1` | 295 |
`yBaseOffset` was previously unknown. It is the first Roi integer (105).
### SensorParams (0x39: `00 0e`) [V 0x14cfcd0, names 0x14c1300]
Read order below. "Version" is the stream class version. This unit is 14.
| Offset | Member offset | Type | Name | Needs ver | Value |
|---|---|---|---|---|---|
| 0x3B | +0x08 | f64 | `metersPerLeftMouseUnit` | any | 1.922e-5 |
| 0x43 | +0x10 | f64 | `metersPerRightMouseUnit` | any | 1.878e-5 |
| 0x4B | +0x00 | f64 | `laserReadoutDelay` | >= 9 | 0.0 |
| 0x53 | +0x18 | f32 | `camToLeftMouseAngle` | >= 14 | 0.0 |
| 0x57 | +0x1C | f64[2] (count 2) | `camToLeftMouseVec` (m) | any | (0.01125, 0.0538) |
| 0x6B | +0x2C | f32 | `camToRightMouseAngle` | >= 14 | 0.0 |
| 0x6F | +0x30 | f64[2] (count 2) | `camToRightMouseVec` (m) | any | (0.03355, -0.00067) |
| (absent) | +0x40 | f64 | `gyroDegPerStep` | >= 15 | **not in this blob** |
| (absent) | +0x48 | f64 | `gyroBias` | >= 15 | **not in this blob** |
So the gyro fields do not exist for this unit: it is a version-14 blob, and the gyro pair needs 15.
The two angles are `float` (setters `movss` 0x14c73e0 / 0x14c7450, getters `fld dword` 0x14c7400 / 0x14c7470) [V].
`imagerGain` and `exposureTimeUs` are ColorParams members, not Sensor members. `yBaseOffset` is in Roi.
### UndistortionParams (0x83: `00 0e`) [V 0x14cefe0, names 0x14c0510]
| Offset | Member offset | Type | Name | Needs ver | Value |
|---|---|---|---|---|---|
| 0x85 | +0x00 | f64 | `focalLength` (px) | any | 737.328 |
| 0x8D | +0x08 | f64 | `principalPointX` (px) | any | 317.546 |
| 0x95 | +0x10 | f64 | `principalPointY` (px) | any | 132.733 |
| 0x9D | +0x18 | i32 | `distModel` (`Utilities::Enums::EDistortionModel`) | >= 8 | 4 |
| 0xA1 | +0x1C | f64[5] (count 5) | `distCoefficients` | any | 0, -0.403169, 1.3805e-4, 2.6490e-3, 0 |
| 0xCD | +0x44 | f64[8] (count 8) | `tiltH` | any | see below |
| 0x111 | +0x84 | f64 | `pixelSize` (m) | any | 6.1305e-5 |
`tiltH` = 1.002744, 0.007833, -0.985882, -0.000164, 1.006565, -0.435809, -1.17e-6, 2.47e-5.
Eight numbers for a 3x3 homography whose ninth entry is implicit (=1). Usage: see "How it is used" below.
### ColorParams (0x119: `00 0e`) [V 0x14cde20, names 0x14bedb0]
The sample sets are `std::vector<std::vector<SamplingPointAbstract<short,unsigned char,3>>>` (member +0x00,
two inner vectors: black, white). Framing, in stream order:
| Offset | Content |
|---|---|
| 0x11B | `00 00` class info of outer vector |
| 0x11D | u32 2 (outer count), u32 0 (item version) |
| 0x125 | `00 00` class info of inner vector |
| 0x127 | u32 827 (count), u32 0 (item version), `00 00` class info of the sample type |
| 0x131 | 827 samples: i16 x, i16 y, `u8[3]` colour (u32 count 3 + b, g, r). The first sample also carries a `00 00` class info for `boost::array` between y and the count (13 bytes); the rest are 11 bytes |
| 0x24BD | u32 873 (count), u32 0 (item version), then 873 samples of 11 bytes (x, y, 3, b, g, r) |
Sample arrays are `SamplingPointAbstract<short, uchar, 3>`: the two coordinates are signed shorts. Scalar members follow:
| Offset | Member offset | Type | Name | Needs ver | Value |
|---|---|---|---|---|---|
| 0x4A47 | +0x0C | u32 | `exposureTimeUs` | >= 8 | 440 |
| 0x4A4B | +0x18 | i32 | `gainR` | >= 8 | 115 |
| 0x4A4F | +0x1C | i32 | `gainG` | >= 8 | 89 |
| 0x4A53 | +0x20 | i32 | `gainB` | >= 8 | 64 |
| 0x4A57 | +0x10 | f64 | `imagerGain` | >= 9 | 1.5 |
| 0x4A5F | +0x24 | f64[3] (count 3) | `lambda` (one per channel) | >= 10 | 1.921e12, 5.193e7, 5.098e7 |
| 0x4A7B | +0x3C | f64 | `lambdaNormFactor` | >= 11 | 6025357260.0 |
| 0x4A83 | +0x44 | i32 | `nurbsModelSize` | >= 12 | 5 |
| 0x4A87 | +0x48 | f32[3] (count 3) | `nurbsSmoothingFactors` | >= 12 | 50, 50, 50 |
| 0x4A97 | +0x54 | i32 | `nurbsNSmoothingIterations` | >= 12 | 100 |
| 0x4A9B | +0x58 | i32[4] (count 4) | `clipOffset` | >= 14 | 8, 12, 8, 12 |
The names are from the property-tree writer 0x14bedb0 (offsets confirm the mapping). Corrections to older notes:
`imagerGain` is the 1.5 (not "lambda"), `lambda` is the 3-vector (not "lambdaNormFactor"), the 6.0e9 double is
`lambdaNormFactor`, the "u32 5" is `nurbsModelSize` (a scalar, not a list), and `4, 8, 12, 8, 12` is the
`clipOffset` int[4] with its count (so clipOffset = 8, 12, 8, 12).
`clipOffset` is read as a `QList<int>` by 0x14c5600 and used in `QualityTrackManager` setup (0x115bc70 [V]): it is subtracted
from the ROI to size the overlap estimator, `height_px = (roiY1 - roiY0) - clip[1] - clip[3]`, `width_px = (roiX1 - roiX0) - clip[0] - clip[2]`
(then times `pixelSize`), so the order is `{left, top, right, bottom}` margins in pixels [G for the edge names, V for the pairing].
The app's built-in default (HWSetup ini, string `8 28 8 12`) has the same shape.
### InfoParams (0x4AAF: `00 0e`) [V 0x14cc1a0, names 0x14c2bc0]
The first 16 bytes of the Info member are a **GUID** (`Data1 u32, Data2 u16, Data3 u16, Data4 u8[8]`; the
array is written with `u32 count 8`). The previous "8 bytes before fingerprint" are Data1..Data3, and the old
"fingerprint" is Data4.
| Offset | Member offset | Type | Name | Needs ver | Value |
|---|---|---|---|---|---|
| 0x4AB1 | +0x00 | u32 | `fingerprint` Data1 | any | 0x84F3094A |
| 0x4AB5 | +0x04 | u16 | Data2 | any | 0x7B88 |
| 0x4AB7 | +0x06 | u16 | Data3 | any | 0x49C6 (UUID version 4) |
| 0x4AB9 | +0x08 | u8[8] (count 8) | Data4 | any | 86 57 3a 6a 38 5e 6f 68 |
| 0x4AC5 | +0x10 | u32 | `calibrationTime` (time_t) | any | 0x5269BEBF = 2013-10-24 23:35:27 UTC |
| 0x4AC9 | +0x14 | f32 | `avgReprojErrorIntrinsic` (px) | any | 0.158 |
| 0x4ACD | +0x18 | f32 | `avgReprojErrorHomography` (px) | any | 0.133 |
| 0x4AD1 | +0x1C | u32 | `product` (`Utilities::Enums::EProduct`; not in the ptree writer) | >= 13 | 4 |
| 0x4AD5 | +0x20 | u32 | `archiveVersion` | >= 11 (else it takes the class version) | 14 |
The old "enum 14" is `archiveVersion`, which equals the class version here. The old "archiveVersion 4" at +0x1C is the
`product` enum: its accessors are 0x14c7880 (set) and 0x14c7890 (get) [V]; `HalLuckyCat::open` (0x1331ae0) copies it into the
driver config (`config+8`) and validates it via 0x1300ff0, which maps it to a profile resource `:/mouse_driver_profiles/<EProduct_*>.xml`.
For archives older than 13 the product reads as 0, and `loadHardwareData` (0x1335430, "CONVERT_LSM200_HACK") rewrites it to 1 for old
units whose `calibrationTime` falls in a window around Jan-May 2012 (compared with `QDate(2012,1,1)` and `QDate(2012,5,1)`) and whose `pixelSize` is below a threshold,
and it also replaces camToLeft/Right vectors with fixed values (0.0141, 0.0525) and (0.036, 0) there [V 0x1335430, exact date logic not worked out]. This unit (2013) is unaffected. `open` picks `DriverFactoryLG` for product in {2, 13, 14}, otherwise
`DriverFactoryLL`; product 4 (this unit) therefore uses `DriverFactoryLL`. The profile resources are named
`EProduct_ADI_WIRED/WIRELESS, ASD_SLAMBASIC/SLAMBASIC2/SLAMGYRO/SLAMLITE/WIRELESS, FOX_WIRED/WIRED_GYRO/WIRELESS, LG_LSM100/150/200/300, Passport, WING_WIRED`; the
numeric order of the enum (moc table) is not readable in the binary, so which product is 4 is [?].
Full UUID: `84f3094a-7b88-49c6-8657-3a6a385e6f68` (the unit's unique id, matches the old "fingerprint").
None of the product/vendor/ODM text strings are in this archive; those live in the `HWSetup` ini struct
(0x1441300: `productString` `EProduct_LG_LSM100`, `vendorString` `EVendor_LG`, `odmString` `EOdm_Chicony`,
`exposureTimeUsMin/Max`, `skewedImageGain`).
### Signature
| Offset | Value | Meaning |
|---|---|---|
| 0x4AD9 | 256 bytes, to the end of the blob | RSA-2048 PKCS#1 v1.5 signature with SHA-1 (DigestInfo `3021300906052b0e03021a05000414`) over blob bytes `[0, N-256)`, i.e. everything before the signature, header excluded [D]. **Verifies** against `calib/calib_rsa.pub.pem` (public exponent 35, an unusual value) with `cryptography`. |
## Integrity
### Checksum (0x3002) [D, matched against the dump; not seen in disassembly]
`CRC-32` (the zlib `crc32()`, polynomial 0xEDB88320, init/xorout 0xFFFFFFFF) over **all N blob bytes including
the 256-byte signature**, stored little-endian. On this unit: `zlib.crc32(blob) == 0x3cd95f42` and the EEPROM bytes at
0x3002 are `42 5f d9 3c`. The sum, Adler-32, SHA-1 and MD5 variants were tried and do not match.
Use in the app (`HalLuckyCat::loadHardwareData`, 0x1335430 [V]): the checksum is read via `readBinaryDataCheckSum`
(0x138d410) and turned into a **cache file name** `<PathsPool::logDir()>/<crc as 8 hex digits>.bin`.
- Checksum 0 or 0xFFFFFFFF (unprogrammed EEPROM): skip the cache and read the blob from the mouse.
- Otherwise the app tries the cache file ("Read HardwareData from file.", parser 0x14ba4e0). On failure it logs "Error reading
HardwareData from file. Try reading from mouse" and reads the blob from the EEPROM (`ovtReadHardwareData`, 0x137e0e0).
After a read from the mouse it writes the cache (0x14bc150).
So the CRC is a cache key; a wrong value would only cause a cache miss or a stale cache hit, not a rejection.
No CRC computation happens on the read path (the CRC-32 table is at 0x1c0c730, used by a zlib routine at 0xeff640).
### Signature [V]
`hal::verifyRSASignature(data, dataLen, sig, sigLen)` is 0x12ff5d0: it loads the PEM public key from the Qt resource
`:/calib_rsa.pub.txt` (`PEM_read_bio_PUBKEY`) and runs `EVP_sha1` + `EVP_DigestUpdate` + `EVP_VerifyFinal`, i.e.
**RSASSA-PKCS1-v1_5 with SHA-1**.
It is called from `ovtReadHardwareData` (0x137e0e0) after `HardwareDataSerializer::fromRawData` (0x14c3720). `fromRawData`
returns the archive stream position after `InfoParams`; that is the length of the signed part. The acceptance rule is:
ok = (archiveVersion < 11) or (0 < consumed < N and verifyRSASignature(blob, consumed, blob + consumed, N - consumed))
So blobs with `archiveVersion` >= 11 must carry a valid signature over `blob[0 .. consumed)` (everything before the
signature; the 6-byte EEPROM header is not included). Older archives (< 11) are accepted unsigned.
This unit: consumed = N - 256 = 19161, the signature is the last 256 bytes, and **it verifies**:
`calib/calib_rsa.pub.pem` (RSA-2048, public exponent 35) recovers
`0001ff..ff00 3021300906052b0e03021a05000414 b5dab6a61406d37057c5381e242b7424f22a952e` and
`SHA1(blob[:19161]) = b5dab6a61406d37057c5381e242b7424f22a952e`. The key is copied to `calib/calib_rsa.pub.pem`.
(Verification only. The private key is not in the app; writes are produced by the factory tool.)
## How the calibration is used
### Undistortion map (`generateMapFromParams`, 0x120e160 [V]; per-pixel kernel 0x10f1750 [V])
The app builds the K matrix and the distortion vector from the Undistortion members and fills two float
maps `mapX`, `mapY` (one entry per *output* pixel). Accessors: focal 0x14c5d00, ppX 0x14c5d10, ppY 0x14c5d20,
`distCoefficients` 0x14c5d90 (raw pointer to member +0x1C), `tiltH` 0x14c5e00 (member +0x44), `pixelSize` 0x14c5e30.
K = [[f, 0, cx], [0, f, cy], [0, 0, 1]] (fx = fy = focalLength)
H = tiltH row-major, 3x3, H[2][2] = 1.0 (h00 h01 h02 h10 h11 h12 h20 h21)
M = inv(K) * inv(H) (two cvInvert, one cvGEMM; OpenCV 2.4.3)
for every output pixel (x, y):
(X, Y, W) = M * (x, y, 1); x' = X/W, y' = Y/W
r2 = x'^2 + y'^2
radial = 1 + c0*r2 + c1*r2^2 + c4*r2^3
xd = x'*radial + 2*c2*x'*y' + c3*(r2 + 2*x'^2)
yd = y'*radial + c2*(r2 + 2*y'^2) + 2*c3*x'*y'
mapX = K00*xd + K01*yd + K02 ; mapY = K10*xd + K11*yd + K12
- `distCoefficients` = OpenCV order `[k1, k2, p1, p2, k3]` ([V] from the kernel: indices 0, 1, 4 radial; 2, 3 tangential).
The unit's values: k1 = 0, **k2 = -0.40317**, p1 = 1.38e-4, p2 = 2.65e-3, k3 = 0. This is the standard OpenCV
5-parameter model (the same code path as OpenCV's `initUndistortRectifyMap` with `newCameraMatrix = H*K` in the sense
`dest = H * K * normalised`).
- `tiltH` is a homography in **pixel space, applied after K**: forward direction is `dest = H * K * (x_d-undistorted
normalised coords)`, which is what `M = inv(K)*inv(H)` encodes. It is *not* a rotation on normalised rays.
For this unit H is close to identity (shift about (-0.99, -0.44) px, scale 1.003 / 1.007, tiny perspective terms).
- `distModel` is **never read** in this build: its getter 0x14c5d40 has no callers (vtable slot only). The
`EDistortionModel` enum has a name map (0x14cadf0) but the map builder always uses the 5-coefficient model above. So
value 4 is cosmetic here [V by absence of callers]. What 4 means in the enum is [?].
- The cvUndistortPoints / cvInitUndistortMap error text "Distortion coefficients must be 1x4, 4x1, 1x5..." (0xb65990,
0xb6db50) is plain OpenCV 2.4.3 code; it takes the same 5-vector.
### Sensor set-up and misc (`HalLuckyCat::open`, 0x1331ae0 [V])
`open` copies the blob into the driver config (`config` = param_2) with the `HardwareData` accessors (`color` 0x14c4ca0 @+0x00,
`undistortion` 0x14c4cb0 @+0x68, `sensor` 0x14c4cd0 @+0xF4, `roi` 0x14c4cf0 @+0x144, `info` 0x14c4d10 @+0x158):
| Blob field | Goes to | Evidence |
|---|---|---|
| `exposureTimeUs` | `config+0x2F0` = float(exposureTimeUs) * 0.001f (so **ms**: 440 -> 0.44 ms) | getter 0x14c5470 called at 0x13328a8; mul at 0x13328ba (constant 0.001f at 0x1cf9a94). This confirms the u32 440 |
| `imagerGain` | `config+0x2F4` (double) | getter 0x14c54a0 at 0x13328d5 |
| `gainR/G/B` | `config+0x2FC/0x300/0x304` | getters 0x14c54d0 / 0x14c54e0 / 0x14c54f0 |
| roi x0, y0, x1, y1 | `config+0x2D4..0x2E0` (defaults 0, 0, 640, 296) | 0x14c7680 |
| `yBaseOffset` | `config+0x2E4` | getter 0x14c76d0 |
| `product` | `config+8` | 0x14c7890 |
The reverse direction (config -> blob, used when the app writes/derives a new calibration) uses the setters 0x14c5460 (exposure,
`int(ms / 0.001)`), 0x14c5480, 0x14c54b0, 0x14c7620, 0x14c76c0, 0x14c7880.
### Colour correction (NURBS surfaces)
The blob stores *samples*, not surfaces. `FUN_011e7950` ([V], 3 passes, one per colour channel `c` = 0, 1, 2) turns the black and the
white sample set into B-spline surfaces (library `UCBspl`: `SplineSurface`, `MGsystem` multigrid, `LSsystem` least squares):
for c in (0, 1, 2):
x[i], y[i] = sample.x, sample.y ; z[i] = sample.colour_byte[c] # sample = 8-byte struct: i16 x, i16 y, u8[3]
sys = MGsystem(1, 1, nurbsModelSize = 5, x, y, z) # 0x11fc2d0: multilevel B-spline, levels grow as 2^k
if nurbsSmoothingFactors[c] != 0:
sys.setSmoothing(nurbsSmoothingFactors[c]) # 0x11fdb70 (double)
sys.iterate(nurbsNSmoothingIterations, 3) # 0x11fda30
surfaces.append(SplineSurface(sys)) # 0x11fd660
So `nurbsModelSize` (5) is the number of refinement levels of the multilevel B-spline, `nurbsSmoothingFactors[c]` is the per-channel
smoothing weight (50 here) and `nurbsNSmoothingIterations` (100) is the iteration count of the smoothing solver; accessors
0x14c55a0 (modelSize), 0x14c55d0 (factors pointer), 0x14c55f0 (iterations) have the single caller 0x11e7950 [V]. The black surface and
the white surface are evaluated per pixel by the colour balancer (`NurbsColorBalancerPolicy`); the linear map
`out = slope*in + intercept` and the target 250 (`colorDestBrightImage`) are in `RE_colour_stitch.md`.
The sample sets themselves are the `nurbsModelBlack/White` members (the vector-of-vectors at ColorParams +0x00).
The three passes use colour-byte index 0, 1, 2 of each sample, which is the stored order b, g, r.
`lambda` / `lambdaNormFactor`: the library message "`LSsystem::calcLambda(): e_norm is zero when calculating lambda`" (string at 0x1b106c4)
suggests [G] that `lambda` is the **regularisation weight computed by the least-squares solver** for a fit
(`LSsystem::calcLambda`, balances data fit against the smoothness term), stored once per channel at calibration time
(1.92e12, 5.19e7, 5.10e7). That is an inference from the library name only; the strings have no direct code reference in the decompilation. `lambdaNormFactor` (6.025e9) normalises it. The only reader is accessor 0x14c5570, which returns
`lambda[0] / lambdaNormFactor` (= 318.9 on this unit) [V]. Its two callers are the tracker set-ups `FUN_011543e0` (0x115460b, `FastTrackLKM`
parameters) and `FUN_0115bc70` (0x115c341, `QualityTrackManager`), where it is passed as one double of the parameter struct next to
`pixelSize`. The factory-line settings key `colorExpectedNormalizedLambda1Min` (read at 0x1422b40, also 0x14800d0) looks like a lower bound
for this normalised value, i.e. a calibration quality gate [G]. What the tracker does with the value is [?]. `lambda[1]` and `lambda[2]`
have no direct accessor call: the pointer accessor 0x14c5530 has only a vtable data reference.
## Open questions
- Read the 64-byte ID at 0x7EB5 from the device (not done; static analysis only so far).
- Which product the `EProduct` value 4 is (the moc enumerator table is not in the binary in readable form).
- The `EDistortionModel` enum value 4 (its name map is built at 0x14cadf0), though the value is never read.
- What the tracker (`FastTrackLKM`, `QualityTrackManager`) does with `lambda[0]/lambdaNormFactor`; `lambda[1]`/`lambda[2]` are never read.
README.md
+7 -2
View File
@@ -5,12 +5,17 @@ Linux driver tools for the Hama myScan (00052343) scanner mouse, a Dacuda design
- `myscan/xu.py` — UVC extension-unit access: switch between mouse and scan mode, LEDs, exposure - `myscan/xu.py` — UVC extension-unit access: switch between mouse and scan mode, LEDs, exposure
- `myscan/frame.py` — fix the lying JPEG header and decode the per-frame sensor record (laser dx/dy) - `myscan/frame.py` — fix the lying JPEG header and decode the per-frame sensor record (laser dx/dy)
- `myscan/stitch.py` — assemble frames into a page image (laser dead reckoning + image registration) - `myscan/calib.py` — per-device calibration from the mouse's EEPROM: colour gains, flat field, lens undistortion
- `myscan/pagegraph.py` — assemble a page from all frames at once: frame-to-frame registration, loop closures
between passes, pose-graph optimisation (used by `scan.py` and `stitch.py`)
- `myscan/stitch.py` — the older live, frame-by-frame stitcher (`stitch.py --live`) and shared helpers
- `tools/scan.py` — scan with the button: `python tools/scan.py` (press to start, press again to stop)
- `tools/record.py` — record a scan session: `python tools/record.py -d 25 -o scan_text` - `tools/record.py` — record a scan session: `python tools/record.py -d 25 -o scan_text`
- `tools/stitch.py` — stitch a recording: `python tools/stitch.py scan_text` -> `scan_text/page.png` - `tools/stitch.py` — stitch a recording: `python tools/stitch.py scan_text` -> `scan_text/page.png`
- `tools/probe.py` — log raw frames and HID reports side by side - `tools/probe.py` — log raw frames and HID reports side by side
- `tools/scantest.py` — minimal scan-mode test: grab frames, decode records, restore mouse mode - `tools/scantest.py` — minimal scan-mode test: grab frames, decode records, restore mouse mode
See [FINDINGS.md](FINDINGS.md) for the protocol and calibration notes. See [FINDINGS.md](FINDINGS.md) for the protocol and calibration notes, and [EEPROM.md](EEPROM.md) for the
calibration blob.
Setup: `python -m venv .venv && .venv/bin/pip install -r requirements.txt` Setup: `python -m venv .venv && .venv/bin/pip install -r requirements.txt`
myscan/calib.py
+27
View File
@@ -13,6 +13,7 @@ white -> colorDestBrightImage = 250).
import os import os
import struct import struct
import cv2
import numpy as np import numpy as np
EEPROM_BASE = 0x3000 EEPROM_BASE = 0x3000
@@ -67,6 +68,8 @@ def parse(data):
"laser_left_m": (d(0x5B), d(0x63)), "laser_left_m": (d(0x5B), d(0x63)),
"laser_right_m": (d(0x73), d(0x7B)), "laser_right_m": (d(0x73), d(0x7B)),
"focal_px": d(0x85), "principal_px": (d(0x8D), d(0x95)), "focal_px": d(0x85), "principal_px": (d(0x8D), d(0x95)),
"dist_coeffs": struct.unpack_from("<5d", blob, 0xA5), # OpenCV k1, k2, p1, p2, k3
"tilt_h": struct.unpack_from("<8d", blob, 0xD1) + (1.0,), # 3x3 row-major, pixel space
"pixel_size_m": d(0x111), "pixel_size_m": d(0x111),
"black": samples[0], "white": samples[1], # columns x, y, b, g, r "black": samples[0], "white": samples[1], # columns x, y, b, g, r
"gains_rgb": struct.unpack_from("<3i", blob, end + 4), # sensor regs 1, 2, 3 "gains_rgb": struct.unpack_from("<3i", blob, end + 4), # sensor regs 1, 2, 3
@@ -125,6 +128,30 @@ class FlatField:
return (f - b) * (DEST_WHITE / np.maximum(w - b, 1)) return (f - b) * (DEST_WHITE / np.maximum(w - b, 1))
class Undistort:
"""Lens + tilt correction, as the app's generateMapFromParams (EEPROM.md): for each
output pixel p, the source pixel is K * distort(inv(K) * inv(H) * p)."""
def __init__(self, cal, shape=(296, 640)):
h, w = shape
f, (cx, cy) = cal["focal_px"], cal["principal_px"]
K = np.array([[f, 0, cx], [0, f, cy], [0, 0, 1.0]])
M = np.linalg.inv(K) @ np.linalg.inv(np.reshape(cal["tilt_h"], (3, 3)))
k1, k2, p1, p2, k3 = cal["dist_coeffs"]
ys, xs = np.mgrid[0:h, 0:w].astype(np.float64)
X, Y, W = (M @ np.stack([xs.ravel(), ys.ravel(), np.ones(xs.size)])).reshape(3, h, w)
x, y = X / W, Y / W
r2 = x * x + y * y
radial = 1 + k1 * r2 + k2 * r2 ** 2 + k3 * r2 ** 3
xd = x * radial + 2 * p1 * x * y + p2 * (r2 + 2 * x * x)
yd = y * radial + p1 * (r2 + 2 * y * y) + 2 * p2 * x * y
self.map = cv2.convertMaps((f * xd + cx).astype(np.float32), (f * yd + cy).astype(np.float32),
cv2.CV_16SC2)
def __call__(self, frame):
return cv2.remap(frame, *self.map, cv2.INTER_LINEAR, borderMode=cv2.BORDER_REPLICATE)
def load(x=None, cache=CACHE): def load(x=None, cache=CACHE):
if x is None and not (cache and os.path.exists(cache)): if x is None and not (cache and os.path.exists(cache)):
from .xu import XU from .xu import XU
myscan/pagegraph.py
+383
View File
@@ -0,0 +1,383 @@
"""Global page assembly: place all frames of a scan at once (pose-graph optimisation).
The live Stitcher places each frame relative to the canvas built so far, so its error
accumulates and a later pass that comes back over earlier content can lock on a wrong
copy. Here every frame keeps its own pose and the poses are solved together:
1. Odometry edges between consecutive frames: laser prediction (with the phase-correlation
override, as in Stitcher), refined by ECC between the two frames when that converges.
After a frame that ECC could not register, the next frame is also registered against
the one before it, which bridges single corrupt frames.
2. Loop-closure edges between non-consecutive keyframes whose current poses overlap:
SIFT matches + RANSAC (rigid), so large drift is recovered, then ECC for sub-pixel.
3. Robust Gauss-Newton over (x, y, theta) per frame. Loop edges that still disagree with
the solution are dropped (repeated content gives false matches) and the solve repeats;
rounds of 2-3 continue with the improved poses so passes far apart find each other.
4. Frames linked to the main page by image-verified edges are blended at their final
pose; the rest (off the page, lifted, corrupt) only had the lasers and are left out,
as are frames that disagree with the mosaic of all the others.
A pose M (3x3) maps frame pixels to canvas pixels; an edge (i, j, T) says M_j ~ M_i @ T.
"""
import os
from concurrent.futures import ThreadPoolExecutor
import numpy as np
import cv2
from scipy.sparse import coo_matrix, diags
from scipy.sparse.linalg import spsolve
from .stitch import (MIN_PHASE_RESP, ROT_GAIN, _aff, _feather, _highpass, compose,
laser_deltas, laser_shift, relative_motion)
ECC_CRIT = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 50, 1e-4)
SIGMA_ECC = 1.0 # px, edge registered by ECC
SIGMA_LASER = 6.0 # px, odometry edge from lasers / phase correlation only
SIGMA_LOOP = 1.5 # px, loop closure
HUBER = 3.0 # robust kernel width, in sigmas
PRUNE = 5.0 # drop loop edges whose residual exceeds this many sigmas
MAX_STEP_ROT = np.radians(2.0) # larger laser rotation between two frames is a miscount
def _rigid(W):
"""2x3 (possibly with a little scale) -> 3x3 rigid transform."""
th = np.arctan2(W[1, 0] - W[0, 1], W[0, 0] + W[1, 1])
return _aff(th, W[0, 2], W[1, 2])
def _ecc(template, image, T):
"""Refine T (image pixels -> template pixels). Returns (T, correlation) or (None, 0)."""
W = np.linalg.inv(T)[:2].astype(np.float32) # ECC warps template coords into image coords
try:
cc, W = cv2.findTransformECC(template, image, W, cv2.MOTION_EUCLIDEAN, ECC_CRIT, None, 5)
except cv2.error:
return None, 0.0
return np.linalg.inv(np.vstack([W, [0, 0, 1]]).astype(np.float64)), cc
def _warp_add(dst, img, M):
"""dst += img warped by M (3x3, img pixels -> dst pixels), touching only the covered box."""
h, w = img.shape[:2]
q = (M @ np.array([[0, 0, 1], [w, 0, 1], [0, h, 1], [w, h, 1.0]]).T)[:2]
x0, y0 = np.maximum(np.floor(q.min(1)).astype(int) - 1, 0)
x1, y1 = np.minimum(np.ceil(q.max(1)).astype(int) + 2, (dst.shape[1], dst.shape[0]))
if x1 > x0 and y1 > y0:
dst[y0:y1, x0:x1] += cv2.warpAffine(img, (_aff(0, -x0, -y0) @ M)[:2], (x1 - x0, y1 - y0),
flags=cv2.INTER_LINEAR)
def _close(A, B, max_deg, max_px):
D = np.linalg.solve(B, A)
return abs(np.degrees(np.arctan2(D[1, 0], D[0, 0]))) <= max_deg and np.hypot(D[0, 2], D[1, 2]) <= max_px
class PageGraph:
def __init__(self, frame_shape, flat=None, undistort=None, rot_gain=ROT_GAIN, min_cc=0.6):
"""flat, undistort: calib.FlatField / calib.Undistort (raw frame -> corrected), or None."""
self.h, self.w = frame_shape[:2]
self.flat, self.undistort, self.rot_gain, self.min_cc = flat, undistort, rot_gain, min_cc
self.center = (self.w / 2, self.h / 2)
self.window = cv2.createHanningWindow((self.w, self.h), cv2.CV_32F)
self.frames, self.hp, self.anchored = [], [], []
self.odo, self.bridges, self.loops = [], [], [] # edges: (i, j, T, sigma)
self.prev_v = None
self.stats = {"frames": 0, "odo_ecc": 0, "odo_laser": 0, "bridges": 0}
@property
def n(self):
return len(self.frames)
def _prep(self, frame):
f = frame.astype(np.float32) if self.flat is None else self.flat(frame)
return f if self.undistort is None else self.undistort(f)
def add(self, frame, v):
"""frame: HxW or HxWx3 uint8 raw frame; v: its record's six raw values."""
f = self._prep(frame)
gray = np.ascontiguousarray(f.mean(axis=2) if f.ndim == 3 else f, np.float32)
hp = _highpass(gray)
v = np.asarray(v)
i = self.n
self.frames.append(frame)
self.hp.append(hp)
self.anchored.append(False)
self.stats["frames"] += 1
if i:
d = laser_deltas(np.stack([self.prev_v, v]))[1]
shift = None
(dx, dy), resp = cv2.phaseCorrelate(self.hp[i - 1], hp, self.window)
if resp > MIN_PHASE_RESP and np.hypot(*(laser_shift(d) - (dx, dy))) > 2:
shift = (dx, dy)
gain = self.rot_gain if abs(self.rot_gain * (d[1] - d[3])) < MAX_STEP_ROT else 0.0
T0 = relative_motion(d, self.center, gain, shift)
T, cc = _ecc(self.hp[i - 1], hp, T0)
if T is not None and cc > self.min_cc and _close(T, T0, 3.0, 15):
self.odo.append((i - 1, i, T, SIGMA_ECC))
self.anchored[i - 1] = self.anchored[i] = True
self.stats["odo_ecc"] += 1
else:
self.odo.append((i - 1, i, T0, SIGMA_LASER))
self.stats["odo_laser"] += 1
if i >= 2 and self.odo[-2][3] != SIGMA_ECC:
Tb0 = self.odo[-2][2] @ self.odo[-1][2]
Tb, cc = _ecc(self.hp[i - 2], hp, Tb0)
if Tb is not None and cc > self.min_cc and _close(Tb, Tb0, 6.0, 60):
self.bridges.append((i - 2, i, Tb, SIGMA_ECC))
self.anchored[i - 2] = self.anchored[i] = True
self.stats["bridges"] += 1
self.prev_v = v
# --- poses -------------------------------------------------------------------
def _chain(self):
P = [np.eye(3)]
for _, _, T, _ in self.odo:
P.append(P[-1] @ T)
return np.array([[M[0, 2], M[1, 2], np.arctan2(M[1, 0], M[0, 0])] for M in P])
def _optimise(self, x, edges, iters=20):
"""Robust Gauss-Newton with frame 0 held fixed. x: (N, 3) -> (x, residual per edge in sigmas)."""
n, E = len(x), len(edges)
a = 150.0 # lever arm of the two residual points
P2 = np.array([[self.center[0] - a, self.center[1]], [self.center[0] + a, self.center[1]]])
I = np.array([e[0] for e in edges])
J = np.array([e[1] for e in edges])
sig = np.array([e[3] for e in edges], np.float64)
Tp = np.einsum("eab,pb->epa", np.array([e[2] for e in edges])[:, :2],
np.hstack([P2, np.ones((2, 1))])) # (E, 2 points, 2): in frame i
Pj = np.broadcast_to(P2, (E, 2, 2))
def rot(c, s, q):
return np.stack([c[:, None] * q[..., 0] - s[:, None] * q[..., 1],
s[:, None] * q[..., 0] + c[:, None] * q[..., 1]], -1)
def drot(c, s, q):
return np.stack([-s[:, None] * q[..., 0] - c[:, None] * q[..., 1],
c[:, None] * q[..., 0] - s[:, None] * q[..., 1]], -1)
pin = np.zeros(n * 3)
pin[:3] = 1e9
rows = np.arange(E * 4).reshape(E, 2, 2)
for _ in range(iters):
ci, si, cj, sj = np.cos(x[I, 2]), np.sin(x[I, 2]), np.cos(x[J, 2]), np.sin(x[J, 2])
r = (rot(cj, sj, Pj) + x[J, None, :2] - rot(ci, si, Tp) - x[I, None, :2]) / sig[:, None, None]
nrm = np.sqrt((r ** 2).sum((1, 2)) / 2)
w = np.where(nrm <= HUBER, 1.0, HUBER / np.maximum(nrm, 1e-9))
dj, di = drot(cj, sj, Pj) / sig[:, None, None], -drot(ci, si, Tp) / sig[:, None, None]
R, C, V = [], [], []
for c in range(2):
for frame, sign, dth in ((J, 1.0, dj), (I, -1.0, di)):
for p in range(2):
R += [rows[:, p, c], rows[:, p, c]]
C += [frame * 3 + c, frame * 3 + 2]
V += [sign / sig, dth[:, p, c]]
Jm = coo_matrix((np.concatenate(V), (np.concatenate(R), np.concatenate(C))),
shape=(E * 4, n * 3)).tocsr()
Wd = diags(np.repeat(w, 4))
H = (Jm.T @ Wd @ Jm + diags(pin + 1e-6)).tocsc()
dx = spsolve(H, -(Jm.T @ (Wd @ r.reshape(-1))))
x = x + dx.reshape(n, 3)
if np.abs(dx).max() < 1e-3:
break
return x, nrm
# --- loop closures ---------------------------------------------------------------
def _keyframes(self, x, step_px=60.0):
keys, last = [], None
for i in range(self.n):
if not self.anchored[i]:
continue
if last is None or np.hypot(*(x[i, :2] - x[last, :2])) > step_px or i - last > 15:
keys.append(i)
last = i
return keys
def _detect(self, i):
g = cv2.normalize(self.hp[i], None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8)
return cv2.SIFT_create(nfeatures=600).detectAndCompute(g, None)
def _features(self, i):
if i not in self._feat:
self._feat[i] = self._detect(i)
return self._feat[i]
def _match(self, a, b):
"""Rigid transform b -> a from features + ECC, or None."""
ka, da = self._features(a)
kb, db = self._features(b)
if da is None or db is None or len(ka) < 15 or len(kb) < 15:
return None
good = [m for m, n2 in (p for p in cv2.BFMatcher(cv2.NORM_L2).knnMatch(db, da, k=2) if len(p) == 2)
if m.distance < 0.75 * n2.distance]
if len(good) < 12:
return None
pb = np.float32([kb[m.queryIdx].pt for m in good])
pa = np.float32([ka[m.trainIdx].pt for m in good])
A, inl = cv2.estimateAffinePartial2D(pb, pa, method=cv2.RANSAC, ransacReprojThreshold=3.0)
if A is None or inl.sum() < 12 or abs(np.hypot(A[0, 0], A[1, 0]) - 1) > 0.05:
return None
T = _rigid(A)
T2, cc = _ecc(self.hp[a], self.hp[b], T)
if T2 is not None and cc > 0.5 and _close(T2, T, 1.0, 4) and self._agree(a, b, T2):
return T2
return None
def _agree(self, a, b, T, min_ncc=0.85, min_tiles=0.8, tile=32):
"""Whole-overlap check. Repeated text (two lines ending in the same words) gives
matches that ECC confirms on the shared words while the rest of the overlap
disagrees: require high NCC over the overlap and agreement in most textured tiles."""
hb = cv2.warpAffine(self.hp[b], T[:2], (self.w, self.h))
m = cv2.warpAffine(np.ones((self.h, self.w), np.float32), T[:2], (self.w, self.h)) > 0.99
m = cv2.erode(m.astype(np.uint8), np.ones((15, 15), np.uint8)) > 0
if m.sum() < 0.05 * m.size or np.corrcoef(self.hp[a][m], hb[m])[0, 1] < min_ncc:
return False
th, tw = self.h // tile, self.w // tile
cut = lambda im: im[:th * tile, :tw * tile].reshape(th, tile, tw, tile).swapaxes(1, 2).reshape(th, tw, -1) # noqa: E731
A, B, M = cut(self.hp[a]), cut(hb), cut(m.astype(np.float32))
A = A - A.mean(-1, keepdims=True)
B = B - B.mean(-1, keepdims=True)
sa, sb = A.std(-1), B.std(-1)
use = (M.mean(-1) > 0.9) & (np.maximum(sa, sb) > 8)
if not use.any():
return True
ncc = (A * B).mean(-1) / np.maximum(sa * sb, 1e-6)
return (ncc[use] > 0.5).mean() >= min_tiles
def _close_loops(self, x, keys, radius, min_gap=12, tries_per_run=3):
"""Match each keyframe against overlapping later keyframes. Candidates are grouped
into runs of consecutive keyframes (one pass over the spot); each run is tried
nearest first and stops at its first match. Runs are matched in parallel."""
P = x[keys, :2]
jobs = []
for ia, a in enumerate(keys):
d = np.hypot(*(P[ia + 1:] - P[ia]).T)
near = [ib for ib in np.nonzero(d < radius)[0] + ia + 1 if keys[ib] - a >= min_gap]
runs, cur = [], []
for ib in near:
if cur and ib - cur[-1] > 2:
runs.append(cur)
cur = []
cur.append(ib)
runs += [cur] if cur else []
for run in runs:
bs = [keys[ib] for ib in sorted(run, key=lambda k: d[k - ia - 1])[:tries_per_run]]
bs = [b for b in bs if (a, b) not in self._tried]
if bs:
jobs.append((a, bs))
def run_job(job):
a, bs = job
for k, b in enumerate(bs):
T = self._match(a, b)
if T is not None:
return (a, b, T, SIGMA_LOOP), k + 1
return None, len(bs)
new = 0
for (a, bs), (edge, tries) in zip(jobs, self._pool.map(run_job, jobs)):
self._tried.update((a, b) for b in bs[:tries])
self.stats["loop_tries"] += tries
if edge is not None:
self.loops.append(edge)
new += 1
return new
def solve(self, rounds=4):
"""Poses (N, 3) after loop closing and optimisation."""
self._feat, self._tried, self.loops = {}, set(), []
self.stats.update(loop_tries=0, loops=0, pruned=0)
x = self._chain()
keys = self._keyframes(x)
self._pool = ThreadPoolExecutor(os.cpu_count())
self._feat.update(zip(keys, self._pool.map(self._detect, keys)))
base = self.odo + self.bridges
for r in range(rounds):
# Wide search first (drift can be hundreds of px), tighter once poses are good.
new = self._close_loops(x, keys, radius=700 if r == 0 else 500)
for _ in range(5):
x, nrm = self._optimise(x, base + self.loops)
bad = nrm[len(base):] > PRUNE
if not bad.any():
break
self.loops = [e for e, b in zip(self.loops, bad) if not b]
self.stats["pruned"] += int(bad.sum())
if r and not new:
break
self.anchored = self._main_component()
self._verify(x)
self._pool.shutdown()
self._pool, self._feat = None, {}
self.stats["loops"] = len(self.loops)
self.stats["rendered"] = int(sum(self.anchored))
self.residuals = nrm
return x
def _main_component(self):
"""Frames linked to the largest group by image-verified edges. Off-page stretches
register frame to frame but connect to the page only through the lasers."""
parent = list(range(self.n))
def find(k):
while parent[k] != k:
parent[k] = parent[parent[k]]
k = parent[k]
return k
for i, j, _, s in self.odo + self.bridges + self.loops:
if s != SIGMA_LASER:
parent[find(i)] = find(j)
roots = np.array([find(k) for k in range(self.n)])
main = np.bincount(roots).argmax()
return list(roots == main)
def _verify(self, x, min_ncc=0.5, scale=0.5):
"""Drop frames that disagree with the mosaic of all frames (a stretch held only by a
wrong registration). Compared on high-passed images at half resolution."""
use = [i for i in range(self.n) if self.anchored[i]]
S = np.diag([scale, scale, 1.0])
Ms, off, W, H = self._layout(x, use, S)
fe = cv2.resize(_feather(self.h, self.w), None, fx=scale, fy=scale)
small = list(self._pool.map(
lambda i: cv2.resize(self.hp[i], None, fx=scale, fy=scale, interpolation=cv2.INTER_AREA), use))
A = [off @ M @ np.linalg.inv(S) for M in Ms]
acc = np.zeros((H, W), np.float32)
ws = np.zeros((H, W), np.float32)
for im, M in zip(small, A):
_warp_add(acc, im * fe, M)
_warp_add(ws, fe, M)
mos = acc / np.maximum(ws, 1e-6)
h, w = fe.shape
def score(k):
back = cv2.warpAffine(mos, np.linalg.inv(A[k])[:2], (w, h))
return np.corrcoef(small[k][10:-10, 10:-10].ravel(), back[10:-10, 10:-10].ravel())[0, 1]
self.ncc = np.full(self.n, np.nan)
self.ncc[use] = list(self._pool.map(score, range(len(use))))
for i in use:
if not self.ncc[i] >= min_ncc:
self.anchored[i] = False
self.stats["inconsistent"] = int(np.sum(self.ncc < min_ncc))
def _layout(self, x, use, S=np.eye(3)):
"""Poses of frames `use` scaled by S, the offset that puts them all on a canvas, and its size."""
Ms = [S @ _aff(x[i, 2], x[i, 0], x[i, 1]) for i in use]
c = np.array([[0, 0, 1], [self.w, 0, 1], [0, self.h, 1], [self.w, self.h, 1.0]]).T
q = np.concatenate([(M @ c)[:2] for M in Ms], 1)
off = _aff(0, 2 - q[0].min(), 2 - q[1].min())
W, H = int(np.ceil(np.ptp(q[0]))) + 4, int(np.ceil(np.ptp(q[1]))) + 4
return Ms, off, W, H
# --- rendering ---------------------------------------------------------------
def image(self, x=None, white_balance=True, background=255):
x = self.solve() if x is None else x
use = [i for i in range(self.n) if self.anchored[i]]
Ms, off, W, H = self._layout(x, use)
ch = self.frames[0].shape[2] if self.frames[0].ndim == 3 else 0
acc = np.zeros((H, W, ch) if ch else (H, W), np.float32)
wsum = np.zeros((H, W), np.float32)
feather = _feather(self.h, self.w)
for i, M in zip(use, Ms):
f = self._prep(self.frames[i])
_warp_add(acc, f * (feather[..., None] if f.ndim == 3 else feather), off @ M)
_warp_add(wsum, feather, off @ M)
return compose(acc, wsum, white_balance, background)
myscan/stitch.py
+9 -4
View File
@@ -177,12 +177,17 @@ class Stitcher:
self.n += 1 self.n += 1
def image(self, white_balance=True, background=255): def image(self, white_balance=True, background=255):
covered = self.wsum > 0.05 return compose(self.acc, self.wsum, white_balance, background)
ws = np.maximum(self.wsum, 1e-6)
out = self.acc / (ws[..., None] if self.ch > 1 else ws)
def compose(acc, wsum, white_balance=True, background=255):
"""Weighted canvas (acc, wsum) -> cropped page image in page orientation."""
covered = wsum > 0.05
ws = np.maximum(wsum, 1e-6)
out = acc / (ws[..., None] if acc.ndim == 3 else ws)
if white_balance: if white_balance:
out = white_patch(out, covered) out = white_patch(out, covered)
m = covered[..., None] if self.ch > 1 else covered m = covered[..., None] if acc.ndim == 3 else covered
out = np.where(m, out, background).clip(0, 255).astype(np.uint8) out = np.where(m, out, background).clip(0, 255).astype(np.uint8)
ys, xs = np.nonzero(covered) ys, xs = np.nonzero(covered)
if len(ys): if len(ys):
requirements.txt
+1
View File
@@ -1,2 +1,3 @@
numpy numpy
opencv-python-headless opencv-python-headless
scipy
tools/scan.py
+38 -18
View File
@@ -1,12 +1,15 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Scan with the scan button, like the original app: press it to start, glide the """Scan with the scan button, like the original app: press it to start, glide the
mouse over the page, press it again to stop. The page is stitched live and saved mouse over the page, press it again to stop. Frames are registered while you scan;
as <outdir>/scan-YYYYmmdd-HHMMSS.png; the mouse goes back to mouse mode between after the stop press the page is assembled from all of them at once (see
scans (and on Ctrl-C).""" myscan/pagegraph.py) and saved as <outdir>/scan-YYYYmmdd-HHMMSS.png. The mouse goes
back to mouse mode between scans (and on Ctrl-C)."""
import argparse import argparse
import csv import csv
import os import os
import queue
import sys import sys
import threading
import time import time
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
@@ -17,7 +20,7 @@ quiet_libjpeg()
import cv2 # noqa: E402 import cv2 # noqa: E402
from myscan import calib # noqa: E402 from myscan import calib # noqa: E402
from myscan.stitch import Stitcher # noqa: E402 from myscan.pagegraph import PageGraph # noqa: E402
from myscan.xu import XU # noqa: E402 from myscan.xu import XU # noqa: E402
@@ -34,16 +37,27 @@ def wait_press(x, poll=0.05):
time.sleep(poll) time.sleep(poll)
def scan_page(x, a, frames_dir=None, flat=None, gains=None): def scan_page(x, a, frames_dir=None, flat=None, undistort=None, gains=None):
st, rows = None, [] """Capture until the button is pressed again. Frames are registered (and saved) on a
worker thread so capture never waits; returns the PageGraph, or None."""
g, rows, work = None, [], queue.Queue()
def worker():
while (item := work.get()) is not None:
n, im, v = item
g.add(im, v)
if frames_dir:
cv2.imwrite(os.path.join(frames_dir, f"f{n:05d}.png"), im)
th = None
with ScanSession(a.video, a.target, a.exposure, xu=x, gains=gains) as s: with ScanSession(a.video, a.target, a.exposure, xu=x, gains=gains) as s:
down, last = 0, 0.0 down, last = 0, 0.0
for t, n, im, rec in s.frames(color=not a.gray): for t, n, im, rec in s.frames(color=not a.gray):
if st is None: if g is None:
st = Stitcher(im.shape, refine=not a.no_refine, flat=flat) g = PageGraph(im.shape, flat=flat, undistort=undistort)
st.add(im, rec.v) th = threading.Thread(target=worker, daemon=True)
th.start()
work.put((n, im, rec.v))
if frames_dir: if frames_dir:
cv2.imwrite(os.path.join(frames_dir, f"f{n:05d}.png"), im)
rows.append([round(t, 4), n, rec.counter, *rec.v, rec.b14, rec.b15, rec.b16, rec.flags, s.e6, rows.append([round(t, 4), n, rec.counter, *rec.v, rec.b14, rec.b15, rec.b16, rec.flags, s.e6,
round(float(im.mean()), 1), rec.raw.hex()]) round(float(im.mean()), 1), rec.raw.hex()])
# Stop on a press: two "down" reads in a row (debounce), ignoring the first # Stop on a press: two "down" reads in a row (debounce), ignoring the first
@@ -56,17 +70,19 @@ def scan_page(x, a, frames_dir=None, flat=None, gains=None):
print("max scan time reached", flush=True) print("max scan time reached", flush=True)
break break
if t - last >= 1: if t - last >= 1:
print(f"\r {t:5.1f}s {st.n} frames canvas {st.size[0]}x{st.size[1]} E6={s.e6}", print(f"\r {t:5.1f}s {n + 1} frames ({g.n} registered) E6={s.e6}", end="", flush=True)
end="", flush=True)
last = t last = t
print() print()
if th:
work.put(None)
th.join()
if frames_dir: # same layout as tools/record.py, so tools/stitch.py can replay it if frames_dir: # same layout as tools/record.py, so tools/stitch.py can replay it
with open(os.path.join(frames_dir, "records.csv"), "w", newline="") as fh: with open(os.path.join(frames_dir, "records.csv"), "w", newline="") as fh:
w = csv.writer(fh) w = csv.writer(fh)
w.writerow(["t", "frame", "counter", "v0", "v1", "v2", "v3", "v4", "v5", w.writerow(["t", "frame", "counter", "v0", "v1", "v2", "v3", "v4", "v5",
"b14", "b15", "b16", "flags", "e6", "brightness", "raw"]) "b14", "b15", "b16", "flags", "e6", "brightness", "raw"])
w.writerows(rows) w.writerows(rows)
return st return g
def main(): def main():
@@ -80,7 +96,7 @@ def main():
ap.add_argument("--gray", action="store_true") ap.add_argument("--gray", action="store_true")
ap.add_argument("--no-wb", action="store_true", help="skip white-patch balancing") ap.add_argument("--no-wb", action="store_true", help="skip white-patch balancing")
ap.add_argument("--no-flat", action="store_true", help="skip the EEPROM flat-field correction") ap.add_argument("--no-flat", action="store_true", help="skip the EEPROM flat-field correction")
ap.add_argument("--no-refine", action="store_true", help="lasers only, no image registration") ap.add_argument("--no-undistort", action="store_true", help="skip the EEPROM lens correction")
ap.add_argument("--save-frames", action="store_true", help="also keep raw frames + records.csv per scan") ap.add_argument("--save-frames", action="store_true", help="also keep raw frames + records.csv per scan")
ap.add_argument("--max-time", type=float, default=180, help="stop a scan after this many seconds") ap.add_argument("--max-time", type=float, default=180, help="stop a scan after this many seconds")
a = ap.parse_args() a = ap.parse_args()
@@ -89,6 +105,7 @@ def main():
print("reading calibration from the mouse's EEPROM (cached after the first time)", flush=True) print("reading calibration from the mouse's EEPROM (cached after the first time)", flush=True)
cal = calib.load(x) cal = calib.load(x)
flat = None if a.no_flat else calib.FlatField(cal) flat = None if a.no_flat else calib.FlatField(cal)
undistort = None if a.no_undistort else calib.Undistort(cal)
gains = cal["gains_rgb"] gains = cal["gains_rgb"]
first = True first = True
try: try:
@@ -102,13 +119,16 @@ def main():
frames_dir = os.path.join(a.outdir, name) if a.save_frames else None frames_dir = os.path.join(a.outdir, name) if a.save_frames else None
if frames_dir: if frames_dir:
os.makedirs(frames_dir) os.makedirs(frames_dir)
st = scan_page(x, a, frames_dir, flat, gains) g = scan_page(x, a, frames_dir, flat, undistort, gains)
if st is None or st.n < 2: if g is None or g.n < 2:
print("no frames captured") print("no frames captured")
else: else:
print("assembling the page...", flush=True)
t0 = time.time()
page = g.image(white_balance=not a.no_wb)
path = os.path.join(a.outdir, name + ".png") path = os.path.join(a.outdir, name + ".png")
cv2.imwrite(path, st.image(white_balance=not a.no_wb)) cv2.imwrite(path, page)
print(f"saved {path} ({st.n} frames, {st.stats})", flush=True) print(f"saved {path} ({time.time() - t0:.1f} s; {g.stats})", flush=True)
if a.once: if a.once:
break break
except KeyboardInterrupt: except KeyboardInterrupt:
tools/stitch.py
+10 -4
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Stitch a directory recorded by tools/record.py into a page image.""" """Stitch a directory recorded by tools/record.py (or scan.py --save-frames) into a page image."""
import argparse import argparse
import csv import csv
import os import os
@@ -11,6 +11,7 @@ import cv2 # noqa: E402
import numpy as np # noqa: E402 import numpy as np # noqa: E402
from myscan import calib # noqa: E402 from myscan import calib # noqa: E402
from myscan.pagegraph import PageGraph # noqa: E402
from myscan.stitch import ROT_GAIN, Stitcher # noqa: E402 from myscan.stitch import ROT_GAIN, Stitcher # noqa: E402
@@ -18,9 +19,11 @@ def main():
ap = argparse.ArgumentParser() ap = argparse.ArgumentParser()
ap.add_argument("dir") ap.add_argument("dir")
ap.add_argument("-o", "--out", help="output image (default <dir>/page.png)") ap.add_argument("-o", "--out", help="output image (default <dir>/page.png)")
ap.add_argument("--no-refine", action="store_true", help="lasers only, no image registration") ap.add_argument("--live", action="store_true", help="use the live frame-by-frame stitcher instead of the global one")
ap.add_argument("--no-refine", action="store_true", help="with --live: lasers only, no image registration")
ap.add_argument("--rot-gain", type=float, default=ROT_GAIN) ap.add_argument("--rot-gain", type=float, default=ROT_GAIN)
ap.add_argument("--no-flat", action="store_true", help="skip the EEPROM flat-field correction") ap.add_argument("--no-flat", action="store_true", help="skip the EEPROM flat-field correction")
ap.add_argument("--no-undistort", action="store_true", help="skip the EEPROM lens correction")
ap.add_argument("--no-wb", action="store_true", help="skip white-patch balancing") ap.add_argument("--no-wb", action="store_true", help="skip white-patch balancing")
ap.add_argument("--start", type=int, default=0) ap.add_argument("--start", type=int, default=0)
ap.add_argument("--end", type=int) ap.add_argument("--end", type=int)
@@ -31,12 +34,15 @@ def main():
frames = [int(r["frame"]) for r in rows] frames = [int(r["frame"]) for r in rows]
rng = range(a.start, a.end if a.end is not None else len(rows)) rng = range(a.start, a.end if a.end is not None else len(rows))
flat = None if a.no_flat else calib.FlatField(calib.load()) cal = calib.load()
flat = None if a.no_flat else calib.FlatField(cal)
undistort = None if a.no_undistort else calib.Undistort(cal)
st = None st = None
for i in rng: for i in rng:
im = cv2.imread(os.path.join(a.dir, f"f{frames[i]:05d}.png"), cv2.IMREAD_UNCHANGED) im = cv2.imread(os.path.join(a.dir, f"f{frames[i]:05d}.png"), cv2.IMREAD_UNCHANGED)
if st is None: if st is None:
st = Stitcher(im.shape, refine=not a.no_refine, rot_gain=a.rot_gain, flat=flat) st = (Stitcher(im.shape, refine=not a.no_refine, rot_gain=a.rot_gain, flat=flat) if a.live
else PageGraph(im.shape, flat=flat, undistort=undistort, rot_gain=a.rot_gain))
st.add(im, v[i]) st.add(im, v[i])
out = a.out or os.path.join(a.dir, "page.png") out = a.out or os.path.join(a.dir, "page.png")
cv2.imwrite(out, st.image(white_balance=not a.no_wb)) cv2.imwrite(out, st.image(white_balance=not a.no_wb))