diff --git a/EEPROM.md b/EEPROM.md new file mode 100644 index 0000000..c1b5bd0 --- /dev/null +++ b/EEPROM.md @@ -0,0 +1,298 @@ +# EEPROM contents (Hama myScan / Dacuda LuckyCat) + +This is a field-by-field map of the per-device calibration blob, from one unit's dump (`calib/eeprom_3000.bin`). +For the read protocol, see `FINDINGS.md` ("Per-device calibration in EEPROM"). For the app side, see +`RE_colour_stitch.md` section 1.4. + +Status tags: +- **[P]**: parsed by `myscan/calib.py`. +- **[D]**: decoded by hand from the bytes; the value is clear. +- **[G]**: guessed field name or meaning. +- **[?]**: unknown. +- **[V]**: verified in disassembly or decompilation; the function address is given as evidence. + +## EEPROM map + +| Address | Size | Content | +|---|---|---| +| 0x3000 | 2 | u16 LE blob length N (19417 on this unit). Valid if 0 < N <= 20000 (0x4E20) [V 0x138c4e0, read of 0x3000 at 0x138c743, bound check 0x138c7a0]. The write path rejects N >= 0x4FB7 [V 0x138cc90] | +| 0x3002 | 4 | u32 LE standard CRC-32 (zlib polynomial) over the N blob bytes, signature included. Verified in Python against the dump: 0x3cd95f42 [D]. The app **does not recompute or compare it on read**: it is only a cache key (see below) [V 0x1335430] | +| 0x3006 | N | `boost::archive::binary_iarchive` blob (i386 sizes), then a 256-byte RSA signature at its tail | +| 0x7EB5 | 64 | mouse ID: raw 64-byte buffer, handled as a string. Read by `OVMouseManager::readMouseId` (0x138e1e0 [V], read at 0x138e4fa), written by `writeMouseId(const char*)` (0x138db20 [V], write at 0x138dd80). **Not yet read from the device.** The `HalLuckyCat` wrapper is 0x1334a60 | +| < 0x3000 | ? | **The app never touches anything below 0x3000.** A scan of every call to `OVIF_DaEEPROMRead/Write` (the stubs 0x17ff6fa / 0x17ff700) in the main binary gives only the callers 0x138c4e0 (0x3000, 0x3006), 0x138d410 (0x3002), 0x138e1e0 (0x7EB5), 0x138cc90 (`writeBinaryData`: a single write starting at 0x3000 for header + checksum + blob, call at 0x138cf3f; the caller builds the whole buffer, the CRC is not computed there) and 0x138db20 (0x7EB5). UVCControl.framework just wraps the raw read/write (`EEPROMRead:withStartAddr:withLength:`). So whatever is below 0x3000 is firmware-side only [V by absence] | + +Everything the app reads from the EEPROM is therefore: length, checksum, blob and the 64-byte ID. + +## Blob layout + +All offsets below are relative to the blob start (EEPROM 0x3006). The whole blob parses with **no gaps** +(a stand-alone parser consumes exactly bytes 0 .. N-256, then the signature follows). + +Stream = `boost::archive::binary_iarchive`, library version 7 (boost ~1.42), i386 sizes. Each class is written +as: `tracking u8`, `class version u8`, then its members, in the order of `serialize`. +All five classes here have `00 0e` = tracking 0, **class version 14**. Fixed C arrays (`T[n]`) are written as +`u32 count` + elements. `std::vector` is `u32 count`, `u32 item_version`, elements. A class that is +first met inside a container (`std::vector`, `boost::array`) writes its own `00 00` (tracking 0, version 0) once. +The app accepts class versions 7..15 only (else `WrongArchiveVersionException`, "Incompatible archive version!", +[V] 0x14cc1a0 and the other four `serialize` functions). + +Serialize functions ([V], `serialize`): Info 0x14cc1a0, Color 0x14cde20, Undistortion 0x14cefe0, +Sensor 0x14cfcd0, Roi 0x14d0990. Names come from the property-tree writers: Undistortion 0x14c0510, Sensor 0x14c1300, +Roi 0x14c23a0, Info 0x14c2bc0, Color 0x14bedb0 (all [V]); the umbrella `HardwareData` has members +color @0x00, undistortion @0x68, sensor @0xF4, roi @0x144, info @0x158 (0x14be690 [V]). + +### Header + +| Offset | Value | Meaning | +|---|---|---| +| 0x00 | u32 22, `serialization::archive` | boost signature [D] | +| 0x1A | `07 04 04 04 08` | library version 7; sizeof int/long/float/double = 4/4/4/8 [D] | +| 0x1F | u32 1 | endianness check [D] | + +### RoiParams (0x23: `00 0e`) [V 0x14d0990, names 0x14c23a0] + +Five plain i32 in this order, no version conditions. + +| Offset | Member offset | Type | Name | Value | +|---|---|---|---|---| +| 0x25 | +0x00 | i32 | `yBaseOffset` | 105 | +| 0x29 | +0x04 | i32 | `roiX0` | 0 | +| 0x2D | +0x08 | i32 | `roiY0` | 0 | +| 0x31 | +0x0C | i32 | `roiX1` | 640 | +| 0x35 | +0x10 | i32 | `roiY1` | 295 | + +`yBaseOffset` was previously unknown. It is the first Roi integer (105). + +### SensorParams (0x39: `00 0e`) [V 0x14cfcd0, names 0x14c1300] + +Read order below. "Version" is the stream class version. This unit is 14. + +| Offset | Member offset | Type | Name | Needs ver | Value | +|---|---|---|---|---|---| +| 0x3B | +0x08 | f64 | `metersPerLeftMouseUnit` | any | 1.922e-5 | +| 0x43 | +0x10 | f64 | `metersPerRightMouseUnit` | any | 1.878e-5 | +| 0x4B | +0x00 | f64 | `laserReadoutDelay` | >= 9 | 0.0 | +| 0x53 | +0x18 | f32 | `camToLeftMouseAngle` | >= 14 | 0.0 | +| 0x57 | +0x1C | f64[2] (count 2) | `camToLeftMouseVec` (m) | any | (0.01125, 0.0538) | +| 0x6B | +0x2C | f32 | `camToRightMouseAngle` | >= 14 | 0.0 | +| 0x6F | +0x30 | f64[2] (count 2) | `camToRightMouseVec` (m) | any | (0.03355, -0.00067) | +| (absent) | +0x40 | f64 | `gyroDegPerStep` | >= 15 | **not in this blob** | +| (absent) | +0x48 | f64 | `gyroBias` | >= 15 | **not in this blob** | + +So the gyro fields do not exist for this unit: it is a version-14 blob, and the gyro pair needs 15. +The two angles are `float` (setters `movss` 0x14c73e0 / 0x14c7450, getters `fld dword` 0x14c7400 / 0x14c7470) [V]. +`imagerGain` and `exposureTimeUs` are ColorParams members, not Sensor members. `yBaseOffset` is in Roi. + +### UndistortionParams (0x83: `00 0e`) [V 0x14cefe0, names 0x14c0510] + +| Offset | Member offset | Type | Name | Needs ver | Value | +|---|---|---|---|---|---| +| 0x85 | +0x00 | f64 | `focalLength` (px) | any | 737.328 | +| 0x8D | +0x08 | f64 | `principalPointX` (px) | any | 317.546 | +| 0x95 | +0x10 | f64 | `principalPointY` (px) | any | 132.733 | +| 0x9D | +0x18 | i32 | `distModel` (`Utilities::Enums::EDistortionModel`) | >= 8 | 4 | +| 0xA1 | +0x1C | f64[5] (count 5) | `distCoefficients` | any | 0, -0.403169, 1.3805e-4, 2.6490e-3, 0 | +| 0xCD | +0x44 | f64[8] (count 8) | `tiltH` | any | see below | +| 0x111 | +0x84 | f64 | `pixelSize` (m) | any | 6.1305e-5 | + +`tiltH` = 1.002744, 0.007833, -0.985882, -0.000164, 1.006565, -0.435809, -1.17e-6, 2.47e-5. +Eight numbers for a 3x3 homography whose ninth entry is implicit (=1). Usage: see "How it is used" below. + +### ColorParams (0x119: `00 0e`) [V 0x14cde20, names 0x14bedb0] + +The sample sets are `std::vector>>` (member +0x00, +two inner vectors: black, white). Framing, in stream order: + +| Offset | Content | +|---|---| +| 0x11B | `00 00` class info of outer vector | +| 0x11D | u32 2 (outer count), u32 0 (item version) | +| 0x125 | `00 00` class info of inner vector | +| 0x127 | u32 827 (count), u32 0 (item version), `00 00` class info of the sample type | +| 0x131 | 827 samples: i16 x, i16 y, `u8[3]` colour (u32 count 3 + b, g, r). The first sample also carries a `00 00` class info for `boost::array` between y and the count (13 bytes); the rest are 11 bytes | +| 0x24BD | u32 873 (count), u32 0 (item version), then 873 samples of 11 bytes (x, y, 3, b, g, r) | + +Sample arrays are `SamplingPointAbstract`: the two coordinates are signed shorts. Scalar members follow: + +| Offset | Member offset | Type | Name | Needs ver | Value | +|---|---|---|---|---|---| +| 0x4A47 | +0x0C | u32 | `exposureTimeUs` | >= 8 | 440 | +| 0x4A4B | +0x18 | i32 | `gainR` | >= 8 | 115 | +| 0x4A4F | +0x1C | i32 | `gainG` | >= 8 | 89 | +| 0x4A53 | +0x20 | i32 | `gainB` | >= 8 | 64 | +| 0x4A57 | +0x10 | f64 | `imagerGain` | >= 9 | 1.5 | +| 0x4A5F | +0x24 | f64[3] (count 3) | `lambda` (one per channel) | >= 10 | 1.921e12, 5.193e7, 5.098e7 | +| 0x4A7B | +0x3C | f64 | `lambdaNormFactor` | >= 11 | 6025357260.0 | +| 0x4A83 | +0x44 | i32 | `nurbsModelSize` | >= 12 | 5 | +| 0x4A87 | +0x48 | f32[3] (count 3) | `nurbsSmoothingFactors` | >= 12 | 50, 50, 50 | +| 0x4A97 | +0x54 | i32 | `nurbsNSmoothingIterations` | >= 12 | 100 | +| 0x4A9B | +0x58 | i32[4] (count 4) | `clipOffset` | >= 14 | 8, 12, 8, 12 | + +The names are from the property-tree writer 0x14bedb0 (offsets confirm the mapping). Corrections to older notes: +`imagerGain` is the 1.5 (not "lambda"), `lambda` is the 3-vector (not "lambdaNormFactor"), the 6.0e9 double is +`lambdaNormFactor`, the "u32 5" is `nurbsModelSize` (a scalar, not a list), and `4, 8, 12, 8, 12` is the +`clipOffset` int[4] with its count (so clipOffset = 8, 12, 8, 12). +`clipOffset` is read as a `QList` by 0x14c5600 and used in `QualityTrackManager` setup (0x115bc70 [V]): it is subtracted +from the ROI to size the overlap estimator, `height_px = (roiY1 - roiY0) - clip[1] - clip[3]`, `width_px = (roiX1 - roiX0) - clip[0] - clip[2]` +(then times `pixelSize`), so the order is `{left, top, right, bottom}` margins in pixels [G for the edge names, V for the pairing]. +The app's built-in default (HWSetup ini, string `8 28 8 12`) has the same shape. + +### InfoParams (0x4AAF: `00 0e`) [V 0x14cc1a0, names 0x14c2bc0] + +The first 16 bytes of the Info member are a **GUID** (`Data1 u32, Data2 u16, Data3 u16, Data4 u8[8]`; the +array is written with `u32 count 8`). The previous "8 bytes before fingerprint" are Data1..Data3, and the old +"fingerprint" is Data4. + +| Offset | Member offset | Type | Name | Needs ver | Value | +|---|---|---|---|---|---| +| 0x4AB1 | +0x00 | u32 | `fingerprint` Data1 | any | 0x84F3094A | +| 0x4AB5 | +0x04 | u16 | Data2 | any | 0x7B88 | +| 0x4AB7 | +0x06 | u16 | Data3 | any | 0x49C6 (UUID version 4) | +| 0x4AB9 | +0x08 | u8[8] (count 8) | Data4 | any | 86 57 3a 6a 38 5e 6f 68 | +| 0x4AC5 | +0x10 | u32 | `calibrationTime` (time_t) | any | 0x5269BEBF = 2013-10-24 23:35:27 UTC | +| 0x4AC9 | +0x14 | f32 | `avgReprojErrorIntrinsic` (px) | any | 0.158 | +| 0x4ACD | +0x18 | f32 | `avgReprojErrorHomography` (px) | any | 0.133 | +| 0x4AD1 | +0x1C | u32 | `product` (`Utilities::Enums::EProduct`; not in the ptree writer) | >= 13 | 4 | +| 0x4AD5 | +0x20 | u32 | `archiveVersion` | >= 11 (else it takes the class version) | 14 | + +The old "enum 14" is `archiveVersion`, which equals the class version here. The old "archiveVersion 4" at +0x1C is the +`product` enum: its accessors are 0x14c7880 (set) and 0x14c7890 (get) [V]; `HalLuckyCat::open` (0x1331ae0) copies it into the +driver config (`config+8`) and validates it via 0x1300ff0, which maps it to a profile resource `:/mouse_driver_profiles/.xml`. +For archives older than 13 the product reads as 0, and `loadHardwareData` (0x1335430, "CONVERT_LSM200_HACK") rewrites it to 1 for old +units whose `calibrationTime` falls in a window around Jan-May 2012 (compared with `QDate(2012,1,1)` and `QDate(2012,5,1)`) and whose `pixelSize` is below a threshold, +and it also replaces camToLeft/Right vectors with fixed values (0.0141, 0.0525) and (0.036, 0) there [V 0x1335430, exact date logic not worked out]. This unit (2013) is unaffected. `open` picks `DriverFactoryLG` for product in {2, 13, 14}, otherwise +`DriverFactoryLL`; product 4 (this unit) therefore uses `DriverFactoryLL`. The profile resources are named +`EProduct_ADI_WIRED/WIRELESS, ASD_SLAMBASIC/SLAMBASIC2/SLAMGYRO/SLAMLITE/WIRELESS, FOX_WIRED/WIRED_GYRO/WIRELESS, LG_LSM100/150/200/300, Passport, WING_WIRED`; the +numeric order of the enum (moc table) is not readable in the binary, so which product is 4 is [?]. +Full UUID: `84f3094a-7b88-49c6-8657-3a6a385e6f68` (the unit's unique id, matches the old "fingerprint"). +None of the product/vendor/ODM text strings are in this archive; those live in the `HWSetup` ini struct +(0x1441300: `productString` `EProduct_LG_LSM100`, `vendorString` `EVendor_LG`, `odmString` `EOdm_Chicony`, +`exposureTimeUsMin/Max`, `skewedImageGain`). + +### Signature + +| Offset | Value | Meaning | +|---|---|---| +| 0x4AD9 | 256 bytes, to the end of the blob | RSA-2048 PKCS#1 v1.5 signature with SHA-1 (DigestInfo `3021300906052b0e03021a05000414`) over blob bytes `[0, N-256)`, i.e. everything before the signature, header excluded [D]. **Verifies** against `calib/calib_rsa.pub.pem` (public exponent 35, an unusual value) with `cryptography`. | + +## Integrity + +### Checksum (0x3002) [D, matched against the dump; not seen in disassembly] + +`CRC-32` (the zlib `crc32()`, polynomial 0xEDB88320, init/xorout 0xFFFFFFFF) over **all N blob bytes including +the 256-byte signature**, stored little-endian. On this unit: `zlib.crc32(blob) == 0x3cd95f42` and the EEPROM bytes at +0x3002 are `42 5f d9 3c`. The sum, Adler-32, SHA-1 and MD5 variants were tried and do not match. + +Use in the app (`HalLuckyCat::loadHardwareData`, 0x1335430 [V]): the checksum is read via `readBinaryDataCheckSum` +(0x138d410) and turned into a **cache file name** `/.bin`. +- Checksum 0 or 0xFFFFFFFF (unprogrammed EEPROM): skip the cache and read the blob from the mouse. +- Otherwise the app tries the cache file ("Read HardwareData from file.", parser 0x14ba4e0). On failure it logs "Error reading + HardwareData from file. Try reading from mouse" and reads the blob from the EEPROM (`ovtReadHardwareData`, 0x137e0e0). + After a read from the mouse it writes the cache (0x14bc150). +So the CRC is a cache key; a wrong value would only cause a cache miss or a stale cache hit, not a rejection. +No CRC computation happens on the read path (the CRC-32 table is at 0x1c0c730, used by a zlib routine at 0xeff640). + +### Signature [V] + +`hal::verifyRSASignature(data, dataLen, sig, sigLen)` is 0x12ff5d0: it loads the PEM public key from the Qt resource +`:/calib_rsa.pub.txt` (`PEM_read_bio_PUBKEY`) and runs `EVP_sha1` + `EVP_DigestUpdate` + `EVP_VerifyFinal`, i.e. +**RSASSA-PKCS1-v1_5 with SHA-1**. +It is called from `ovtReadHardwareData` (0x137e0e0) after `HardwareDataSerializer::fromRawData` (0x14c3720). `fromRawData` +returns the archive stream position after `InfoParams`; that is the length of the signed part. The acceptance rule is: + + ok = (archiveVersion < 11) or (0 < consumed < N and verifyRSASignature(blob, consumed, blob + consumed, N - consumed)) + +So blobs with `archiveVersion` >= 11 must carry a valid signature over `blob[0 .. consumed)` (everything before the +signature; the 6-byte EEPROM header is not included). Older archives (< 11) are accepted unsigned. +This unit: consumed = N - 256 = 19161, the signature is the last 256 bytes, and **it verifies**: +`calib/calib_rsa.pub.pem` (RSA-2048, public exponent 35) recovers +`0001ff..ff00 3021300906052b0e03021a05000414 b5dab6a61406d37057c5381e242b7424f22a952e` and +`SHA1(blob[:19161]) = b5dab6a61406d37057c5381e242b7424f22a952e`. The key is copied to `calib/calib_rsa.pub.pem`. +(Verification only. The private key is not in the app; writes are produced by the factory tool.) + +## How the calibration is used + +### Undistortion map (`generateMapFromParams`, 0x120e160 [V]; per-pixel kernel 0x10f1750 [V]) + +The app builds the K matrix and the distortion vector from the Undistortion members and fills two float +maps `mapX`, `mapY` (one entry per *output* pixel). Accessors: focal 0x14c5d00, ppX 0x14c5d10, ppY 0x14c5d20, +`distCoefficients` 0x14c5d90 (raw pointer to member +0x1C), `tiltH` 0x14c5e00 (member +0x44), `pixelSize` 0x14c5e30. + + K = [[f, 0, cx], [0, f, cy], [0, 0, 1]] (fx = fy = focalLength) + H = tiltH row-major, 3x3, H[2][2] = 1.0 (h00 h01 h02 h10 h11 h12 h20 h21) + M = inv(K) * inv(H) (two cvInvert, one cvGEMM; OpenCV 2.4.3) + for every output pixel (x, y): + (X, Y, W) = M * (x, y, 1); x' = X/W, y' = Y/W + r2 = x'^2 + y'^2 + radial = 1 + c0*r2 + c1*r2^2 + c4*r2^3 + xd = x'*radial + 2*c2*x'*y' + c3*(r2 + 2*x'^2) + yd = y'*radial + c2*(r2 + 2*y'^2) + 2*c3*x'*y' + mapX = K00*xd + K01*yd + K02 ; mapY = K10*xd + K11*yd + K12 + +- `distCoefficients` = OpenCV order `[k1, k2, p1, p2, k3]` ([V] from the kernel: indices 0, 1, 4 radial; 2, 3 tangential). + The unit's values: k1 = 0, **k2 = -0.40317**, p1 = 1.38e-4, p2 = 2.65e-3, k3 = 0. This is the standard OpenCV + 5-parameter model (the same code path as OpenCV's `initUndistortRectifyMap` with `newCameraMatrix = H*K` in the sense + `dest = H * K * normalised`). +- `tiltH` is a homography in **pixel space, applied after K**: forward direction is `dest = H * K * (x_d-undistorted + normalised coords)`, which is what `M = inv(K)*inv(H)` encodes. It is *not* a rotation on normalised rays. + For this unit H is close to identity (shift about (-0.99, -0.44) px, scale 1.003 / 1.007, tiny perspective terms). +- `distModel` is **never read** in this build: its getter 0x14c5d40 has no callers (vtable slot only). The + `EDistortionModel` enum has a name map (0x14cadf0) but the map builder always uses the 5-coefficient model above. So + value 4 is cosmetic here [V by absence of callers]. What 4 means in the enum is [?]. +- The cvUndistortPoints / cvInitUndistortMap error text "Distortion coefficients must be 1x4, 4x1, 1x5..." (0xb65990, + 0xb6db50) is plain OpenCV 2.4.3 code; it takes the same 5-vector. + +### Sensor set-up and misc (`HalLuckyCat::open`, 0x1331ae0 [V]) + +`open` copies the blob into the driver config (`config` = param_2) with the `HardwareData` accessors (`color` 0x14c4ca0 @+0x00, +`undistortion` 0x14c4cb0 @+0x68, `sensor` 0x14c4cd0 @+0xF4, `roi` 0x14c4cf0 @+0x144, `info` 0x14c4d10 @+0x158): + +| Blob field | Goes to | Evidence | +|---|---|---| +| `exposureTimeUs` | `config+0x2F0` = float(exposureTimeUs) * 0.001f (so **ms**: 440 -> 0.44 ms) | getter 0x14c5470 called at 0x13328a8; mul at 0x13328ba (constant 0.001f at 0x1cf9a94). This confirms the u32 440 | +| `imagerGain` | `config+0x2F4` (double) | getter 0x14c54a0 at 0x13328d5 | +| `gainR/G/B` | `config+0x2FC/0x300/0x304` | getters 0x14c54d0 / 0x14c54e0 / 0x14c54f0 | +| roi x0, y0, x1, y1 | `config+0x2D4..0x2E0` (defaults 0, 0, 640, 296) | 0x14c7680 | +| `yBaseOffset` | `config+0x2E4` | getter 0x14c76d0 | +| `product` | `config+8` | 0x14c7890 | + +The reverse direction (config -> blob, used when the app writes/derives a new calibration) uses the setters 0x14c5460 (exposure, +`int(ms / 0.001)`), 0x14c5480, 0x14c54b0, 0x14c7620, 0x14c76c0, 0x14c7880. + +### Colour correction (NURBS surfaces) + +The blob stores *samples*, not surfaces. `FUN_011e7950` ([V], 3 passes, one per colour channel `c` = 0, 1, 2) turns the black and the +white sample set into B-spline surfaces (library `UCBspl`: `SplineSurface`, `MGsystem` multigrid, `LSsystem` least squares): + + for c in (0, 1, 2): + x[i], y[i] = sample.x, sample.y ; z[i] = sample.colour_byte[c] # sample = 8-byte struct: i16 x, i16 y, u8[3] + sys = MGsystem(1, 1, nurbsModelSize = 5, x, y, z) # 0x11fc2d0: multilevel B-spline, levels grow as 2^k + if nurbsSmoothingFactors[c] != 0: + sys.setSmoothing(nurbsSmoothingFactors[c]) # 0x11fdb70 (double) + sys.iterate(nurbsNSmoothingIterations, 3) # 0x11fda30 + surfaces.append(SplineSurface(sys)) # 0x11fd660 + +So `nurbsModelSize` (5) is the number of refinement levels of the multilevel B-spline, `nurbsSmoothingFactors[c]` is the per-channel +smoothing weight (50 here) and `nurbsNSmoothingIterations` (100) is the iteration count of the smoothing solver; accessors +0x14c55a0 (modelSize), 0x14c55d0 (factors pointer), 0x14c55f0 (iterations) have the single caller 0x11e7950 [V]. The black surface and +the white surface are evaluated per pixel by the colour balancer (`NurbsColorBalancerPolicy`); the linear map +`out = slope*in + intercept` and the target 250 (`colorDestBrightImage`) are in `RE_colour_stitch.md`. +The sample sets themselves are the `nurbsModelBlack/White` members (the vector-of-vectors at ColorParams +0x00). +The three passes use colour-byte index 0, 1, 2 of each sample, which is the stored order b, g, r. + +`lambda` / `lambdaNormFactor`: the library message "`LSsystem::calcLambda(): e_norm is zero when calculating lambda`" (string at 0x1b106c4) +suggests [G] that `lambda` is the **regularisation weight computed by the least-squares solver** for a fit +(`LSsystem::calcLambda`, balances data fit against the smoothness term), stored once per channel at calibration time +(1.92e12, 5.19e7, 5.10e7). That is an inference from the library name only; the strings have no direct code reference in the decompilation. `lambdaNormFactor` (6.025e9) normalises it. The only reader is accessor 0x14c5570, which returns +`lambda[0] / lambdaNormFactor` (= 318.9 on this unit) [V]. Its two callers are the tracker set-ups `FUN_011543e0` (0x115460b, `FastTrackLKM` +parameters) and `FUN_0115bc70` (0x115c341, `QualityTrackManager`), where it is passed as one double of the parameter struct next to +`pixelSize`. The factory-line settings key `colorExpectedNormalizedLambda1Min` (read at 0x1422b40, also 0x14800d0) looks like a lower bound +for this normalised value, i.e. a calibration quality gate [G]. What the tracker does with the value is [?]. `lambda[1]` and `lambda[2]` +have no direct accessor call: the pointer accessor 0x14c5530 has only a vtable data reference. + +## Open questions + +- Read the 64-byte ID at 0x7EB5 from the device (not done; static analysis only so far). +- Which product the `EProduct` value 4 is (the moc enumerator table is not in the binary in readable form). +- The `EDistortionModel` enum value 4 (its name map is built at 0x14cadf0), though the value is never read. +- What the tracker (`FastTrackLKM`, `QualityTrackManager`) does with `lambda[0]/lambdaNormFactor`; `lambda[1]`/`lambda[2]` are never read. diff --git a/README.md b/README.md index 6a45303..c8cffad 100644 --- a/README.md +++ b/README.md @@ -5,12 +5,17 @@ Linux driver tools for the Hama myScan (00052343) scanner mouse, a Dacuda design - `myscan/xu.py` — UVC extension-unit access: switch between mouse and scan mode, LEDs, exposure - `myscan/frame.py` — fix the lying JPEG header and decode the per-frame sensor record (laser dx/dy) -- `myscan/stitch.py` — assemble frames into a page image (laser dead reckoning + image registration) +- `myscan/calib.py` — per-device calibration from the mouse's EEPROM: colour gains, flat field, lens undistortion +- `myscan/pagegraph.py` — assemble a page from all frames at once: frame-to-frame registration, loop closures + between passes, pose-graph optimisation (used by `scan.py` and `stitch.py`) +- `myscan/stitch.py` — the older live, frame-by-frame stitcher (`stitch.py --live`) and shared helpers +- `tools/scan.py` — scan with the button: `python tools/scan.py` (press to start, press again to stop) - `tools/record.py` — record a scan session: `python tools/record.py -d 25 -o scan_text` - `tools/stitch.py` — stitch a recording: `python tools/stitch.py scan_text` -> `scan_text/page.png` - `tools/probe.py` — log raw frames and HID reports side by side - `tools/scantest.py` — minimal scan-mode test: grab frames, decode records, restore mouse mode -See [FINDINGS.md](FINDINGS.md) for the protocol and calibration notes. +See [FINDINGS.md](FINDINGS.md) for the protocol and calibration notes, and [EEPROM.md](EEPROM.md) for the +calibration blob. Setup: `python -m venv .venv && .venv/bin/pip install -r requirements.txt` diff --git a/myscan/calib.py b/myscan/calib.py index 338fcff..7baa565 100644 --- a/myscan/calib.py +++ b/myscan/calib.py @@ -13,6 +13,7 @@ white -> colorDestBrightImage = 250). import os import struct +import cv2 import numpy as np EEPROM_BASE = 0x3000 @@ -67,6 +68,8 @@ def parse(data): "laser_left_m": (d(0x5B), d(0x63)), "laser_right_m": (d(0x73), d(0x7B)), "focal_px": d(0x85), "principal_px": (d(0x8D), d(0x95)), + "dist_coeffs": struct.unpack_from("<5d", blob, 0xA5), # OpenCV k1, k2, p1, p2, k3 + "tilt_h": struct.unpack_from("<8d", blob, 0xD1) + (1.0,), # 3x3 row-major, pixel space "pixel_size_m": d(0x111), "black": samples[0], "white": samples[1], # columns x, y, b, g, r "gains_rgb": struct.unpack_from("<3i", blob, end + 4), # sensor regs 1, 2, 3 @@ -125,6 +128,30 @@ class FlatField: return (f - b) * (DEST_WHITE / np.maximum(w - b, 1)) +class Undistort: + """Lens + tilt correction, as the app's generateMapFromParams (EEPROM.md): for each + output pixel p, the source pixel is K * distort(inv(K) * inv(H) * p).""" + + def __init__(self, cal, shape=(296, 640)): + h, w = shape + f, (cx, cy) = cal["focal_px"], cal["principal_px"] + K = np.array([[f, 0, cx], [0, f, cy], [0, 0, 1.0]]) + M = np.linalg.inv(K) @ np.linalg.inv(np.reshape(cal["tilt_h"], (3, 3))) + k1, k2, p1, p2, k3 = cal["dist_coeffs"] + ys, xs = np.mgrid[0:h, 0:w].astype(np.float64) + X, Y, W = (M @ np.stack([xs.ravel(), ys.ravel(), np.ones(xs.size)])).reshape(3, h, w) + x, y = X / W, Y / W + r2 = x * x + y * y + radial = 1 + k1 * r2 + k2 * r2 ** 2 + k3 * r2 ** 3 + xd = x * radial + 2 * p1 * x * y + p2 * (r2 + 2 * x * x) + yd = y * radial + p1 * (r2 + 2 * y * y) + 2 * p2 * x * y + self.map = cv2.convertMaps((f * xd + cx).astype(np.float32), (f * yd + cy).astype(np.float32), + cv2.CV_16SC2) + + def __call__(self, frame): + return cv2.remap(frame, *self.map, cv2.INTER_LINEAR, borderMode=cv2.BORDER_REPLICATE) + + def load(x=None, cache=CACHE): if x is None and not (cache and os.path.exists(cache)): from .xu import XU diff --git a/myscan/pagegraph.py b/myscan/pagegraph.py new file mode 100644 index 0000000..3aa169e --- /dev/null +++ b/myscan/pagegraph.py @@ -0,0 +1,383 @@ +"""Global page assembly: place all frames of a scan at once (pose-graph optimisation). + +The live Stitcher places each frame relative to the canvas built so far, so its error +accumulates and a later pass that comes back over earlier content can lock on a wrong +copy. Here every frame keeps its own pose and the poses are solved together: + +1. Odometry edges between consecutive frames: laser prediction (with the phase-correlation + override, as in Stitcher), refined by ECC between the two frames when that converges. + After a frame that ECC could not register, the next frame is also registered against + the one before it, which bridges single corrupt frames. +2. Loop-closure edges between non-consecutive keyframes whose current poses overlap: + SIFT matches + RANSAC (rigid), so large drift is recovered, then ECC for sub-pixel. +3. Robust Gauss-Newton over (x, y, theta) per frame. Loop edges that still disagree with + the solution are dropped (repeated content gives false matches) and the solve repeats; + rounds of 2-3 continue with the improved poses so passes far apart find each other. +4. Frames linked to the main page by image-verified edges are blended at their final + pose; the rest (off the page, lifted, corrupt) only had the lasers and are left out, + as are frames that disagree with the mosaic of all the others. + +A pose M (3x3) maps frame pixels to canvas pixels; an edge (i, j, T) says M_j ~ M_i @ T. +""" +import os +from concurrent.futures import ThreadPoolExecutor + +import numpy as np +import cv2 +from scipy.sparse import coo_matrix, diags +from scipy.sparse.linalg import spsolve + +from .stitch import (MIN_PHASE_RESP, ROT_GAIN, _aff, _feather, _highpass, compose, + laser_deltas, laser_shift, relative_motion) + +ECC_CRIT = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 50, 1e-4) +SIGMA_ECC = 1.0 # px, edge registered by ECC +SIGMA_LASER = 6.0 # px, odometry edge from lasers / phase correlation only +SIGMA_LOOP = 1.5 # px, loop closure +HUBER = 3.0 # robust kernel width, in sigmas +PRUNE = 5.0 # drop loop edges whose residual exceeds this many sigmas +MAX_STEP_ROT = np.radians(2.0) # larger laser rotation between two frames is a miscount + + +def _rigid(W): + """2x3 (possibly with a little scale) -> 3x3 rigid transform.""" + th = np.arctan2(W[1, 0] - W[0, 1], W[0, 0] + W[1, 1]) + return _aff(th, W[0, 2], W[1, 2]) + + +def _ecc(template, image, T): + """Refine T (image pixels -> template pixels). Returns (T, correlation) or (None, 0).""" + W = np.linalg.inv(T)[:2].astype(np.float32) # ECC warps template coords into image coords + try: + cc, W = cv2.findTransformECC(template, image, W, cv2.MOTION_EUCLIDEAN, ECC_CRIT, None, 5) + except cv2.error: + return None, 0.0 + return np.linalg.inv(np.vstack([W, [0, 0, 1]]).astype(np.float64)), cc + + +def _warp_add(dst, img, M): + """dst += img warped by M (3x3, img pixels -> dst pixels), touching only the covered box.""" + h, w = img.shape[:2] + q = (M @ np.array([[0, 0, 1], [w, 0, 1], [0, h, 1], [w, h, 1.0]]).T)[:2] + x0, y0 = np.maximum(np.floor(q.min(1)).astype(int) - 1, 0) + x1, y1 = np.minimum(np.ceil(q.max(1)).astype(int) + 2, (dst.shape[1], dst.shape[0])) + if x1 > x0 and y1 > y0: + dst[y0:y1, x0:x1] += cv2.warpAffine(img, (_aff(0, -x0, -y0) @ M)[:2], (x1 - x0, y1 - y0), + flags=cv2.INTER_LINEAR) + + +def _close(A, B, max_deg, max_px): + D = np.linalg.solve(B, A) + return abs(np.degrees(np.arctan2(D[1, 0], D[0, 0]))) <= max_deg and np.hypot(D[0, 2], D[1, 2]) <= max_px + + +class PageGraph: + def __init__(self, frame_shape, flat=None, undistort=None, rot_gain=ROT_GAIN, min_cc=0.6): + """flat, undistort: calib.FlatField / calib.Undistort (raw frame -> corrected), or None.""" + self.h, self.w = frame_shape[:2] + self.flat, self.undistort, self.rot_gain, self.min_cc = flat, undistort, rot_gain, min_cc + self.center = (self.w / 2, self.h / 2) + self.window = cv2.createHanningWindow((self.w, self.h), cv2.CV_32F) + self.frames, self.hp, self.anchored = [], [], [] + self.odo, self.bridges, self.loops = [], [], [] # edges: (i, j, T, sigma) + self.prev_v = None + self.stats = {"frames": 0, "odo_ecc": 0, "odo_laser": 0, "bridges": 0} + + @property + def n(self): + return len(self.frames) + + def _prep(self, frame): + f = frame.astype(np.float32) if self.flat is None else self.flat(frame) + return f if self.undistort is None else self.undistort(f) + + def add(self, frame, v): + """frame: HxW or HxWx3 uint8 raw frame; v: its record's six raw values.""" + f = self._prep(frame) + gray = np.ascontiguousarray(f.mean(axis=2) if f.ndim == 3 else f, np.float32) + hp = _highpass(gray) + v = np.asarray(v) + i = self.n + self.frames.append(frame) + self.hp.append(hp) + self.anchored.append(False) + self.stats["frames"] += 1 + if i: + d = laser_deltas(np.stack([self.prev_v, v]))[1] + shift = None + (dx, dy), resp = cv2.phaseCorrelate(self.hp[i - 1], hp, self.window) + if resp > MIN_PHASE_RESP and np.hypot(*(laser_shift(d) - (dx, dy))) > 2: + shift = (dx, dy) + gain = self.rot_gain if abs(self.rot_gain * (d[1] - d[3])) < MAX_STEP_ROT else 0.0 + T0 = relative_motion(d, self.center, gain, shift) + T, cc = _ecc(self.hp[i - 1], hp, T0) + if T is not None and cc > self.min_cc and _close(T, T0, 3.0, 15): + self.odo.append((i - 1, i, T, SIGMA_ECC)) + self.anchored[i - 1] = self.anchored[i] = True + self.stats["odo_ecc"] += 1 + else: + self.odo.append((i - 1, i, T0, SIGMA_LASER)) + self.stats["odo_laser"] += 1 + if i >= 2 and self.odo[-2][3] != SIGMA_ECC: + Tb0 = self.odo[-2][2] @ self.odo[-1][2] + Tb, cc = _ecc(self.hp[i - 2], hp, Tb0) + if Tb is not None and cc > self.min_cc and _close(Tb, Tb0, 6.0, 60): + self.bridges.append((i - 2, i, Tb, SIGMA_ECC)) + self.anchored[i - 2] = self.anchored[i] = True + self.stats["bridges"] += 1 + self.prev_v = v + + # --- poses ------------------------------------------------------------------- + + def _chain(self): + P = [np.eye(3)] + for _, _, T, _ in self.odo: + P.append(P[-1] @ T) + return np.array([[M[0, 2], M[1, 2], np.arctan2(M[1, 0], M[0, 0])] for M in P]) + + def _optimise(self, x, edges, iters=20): + """Robust Gauss-Newton with frame 0 held fixed. x: (N, 3) -> (x, residual per edge in sigmas).""" + n, E = len(x), len(edges) + a = 150.0 # lever arm of the two residual points + P2 = np.array([[self.center[0] - a, self.center[1]], [self.center[0] + a, self.center[1]]]) + I = np.array([e[0] for e in edges]) + J = np.array([e[1] for e in edges]) + sig = np.array([e[3] for e in edges], np.float64) + Tp = np.einsum("eab,pb->epa", np.array([e[2] for e in edges])[:, :2], + np.hstack([P2, np.ones((2, 1))])) # (E, 2 points, 2): in frame i + Pj = np.broadcast_to(P2, (E, 2, 2)) + + def rot(c, s, q): + return np.stack([c[:, None] * q[..., 0] - s[:, None] * q[..., 1], + s[:, None] * q[..., 0] + c[:, None] * q[..., 1]], -1) + + def drot(c, s, q): + return np.stack([-s[:, None] * q[..., 0] - c[:, None] * q[..., 1], + c[:, None] * q[..., 0] - s[:, None] * q[..., 1]], -1) + + pin = np.zeros(n * 3) + pin[:3] = 1e9 + rows = np.arange(E * 4).reshape(E, 2, 2) + for _ in range(iters): + ci, si, cj, sj = np.cos(x[I, 2]), np.sin(x[I, 2]), np.cos(x[J, 2]), np.sin(x[J, 2]) + r = (rot(cj, sj, Pj) + x[J, None, :2] - rot(ci, si, Tp) - x[I, None, :2]) / sig[:, None, None] + nrm = np.sqrt((r ** 2).sum((1, 2)) / 2) + w = np.where(nrm <= HUBER, 1.0, HUBER / np.maximum(nrm, 1e-9)) + dj, di = drot(cj, sj, Pj) / sig[:, None, None], -drot(ci, si, Tp) / sig[:, None, None] + R, C, V = [], [], [] + for c in range(2): + for frame, sign, dth in ((J, 1.0, dj), (I, -1.0, di)): + for p in range(2): + R += [rows[:, p, c], rows[:, p, c]] + C += [frame * 3 + c, frame * 3 + 2] + V += [sign / sig, dth[:, p, c]] + Jm = coo_matrix((np.concatenate(V), (np.concatenate(R), np.concatenate(C))), + shape=(E * 4, n * 3)).tocsr() + Wd = diags(np.repeat(w, 4)) + H = (Jm.T @ Wd @ Jm + diags(pin + 1e-6)).tocsc() + dx = spsolve(H, -(Jm.T @ (Wd @ r.reshape(-1)))) + x = x + dx.reshape(n, 3) + if np.abs(dx).max() < 1e-3: + break + return x, nrm + + # --- loop closures --------------------------------------------------------------- + + def _keyframes(self, x, step_px=60.0): + keys, last = [], None + for i in range(self.n): + if not self.anchored[i]: + continue + if last is None or np.hypot(*(x[i, :2] - x[last, :2])) > step_px or i - last > 15: + keys.append(i) + last = i + return keys + + def _detect(self, i): + g = cv2.normalize(self.hp[i], None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8) + return cv2.SIFT_create(nfeatures=600).detectAndCompute(g, None) + + def _features(self, i): + if i not in self._feat: + self._feat[i] = self._detect(i) + return self._feat[i] + + def _match(self, a, b): + """Rigid transform b -> a from features + ECC, or None.""" + ka, da = self._features(a) + kb, db = self._features(b) + if da is None or db is None or len(ka) < 15 or len(kb) < 15: + return None + good = [m for m, n2 in (p for p in cv2.BFMatcher(cv2.NORM_L2).knnMatch(db, da, k=2) if len(p) == 2) + if m.distance < 0.75 * n2.distance] + if len(good) < 12: + return None + pb = np.float32([kb[m.queryIdx].pt for m in good]) + pa = np.float32([ka[m.trainIdx].pt for m in good]) + A, inl = cv2.estimateAffinePartial2D(pb, pa, method=cv2.RANSAC, ransacReprojThreshold=3.0) + if A is None or inl.sum() < 12 or abs(np.hypot(A[0, 0], A[1, 0]) - 1) > 0.05: + return None + T = _rigid(A) + T2, cc = _ecc(self.hp[a], self.hp[b], T) + if T2 is not None and cc > 0.5 and _close(T2, T, 1.0, 4) and self._agree(a, b, T2): + return T2 + return None + + def _agree(self, a, b, T, min_ncc=0.85, min_tiles=0.8, tile=32): + """Whole-overlap check. Repeated text (two lines ending in the same words) gives + matches that ECC confirms on the shared words while the rest of the overlap + disagrees: require high NCC over the overlap and agreement in most textured tiles.""" + hb = cv2.warpAffine(self.hp[b], T[:2], (self.w, self.h)) + m = cv2.warpAffine(np.ones((self.h, self.w), np.float32), T[:2], (self.w, self.h)) > 0.99 + m = cv2.erode(m.astype(np.uint8), np.ones((15, 15), np.uint8)) > 0 + if m.sum() < 0.05 * m.size or np.corrcoef(self.hp[a][m], hb[m])[0, 1] < min_ncc: + return False + th, tw = self.h // tile, self.w // tile + cut = lambda im: im[:th * tile, :tw * tile].reshape(th, tile, tw, tile).swapaxes(1, 2).reshape(th, tw, -1) # noqa: E731 + A, B, M = cut(self.hp[a]), cut(hb), cut(m.astype(np.float32)) + A = A - A.mean(-1, keepdims=True) + B = B - B.mean(-1, keepdims=True) + sa, sb = A.std(-1), B.std(-1) + use = (M.mean(-1) > 0.9) & (np.maximum(sa, sb) > 8) + if not use.any(): + return True + ncc = (A * B).mean(-1) / np.maximum(sa * sb, 1e-6) + return (ncc[use] > 0.5).mean() >= min_tiles + + def _close_loops(self, x, keys, radius, min_gap=12, tries_per_run=3): + """Match each keyframe against overlapping later keyframes. Candidates are grouped + into runs of consecutive keyframes (one pass over the spot); each run is tried + nearest first and stops at its first match. Runs are matched in parallel.""" + P = x[keys, :2] + jobs = [] + for ia, a in enumerate(keys): + d = np.hypot(*(P[ia + 1:] - P[ia]).T) + near = [ib for ib in np.nonzero(d < radius)[0] + ia + 1 if keys[ib] - a >= min_gap] + runs, cur = [], [] + for ib in near: + if cur and ib - cur[-1] > 2: + runs.append(cur) + cur = [] + cur.append(ib) + runs += [cur] if cur else [] + for run in runs: + bs = [keys[ib] for ib in sorted(run, key=lambda k: d[k - ia - 1])[:tries_per_run]] + bs = [b for b in bs if (a, b) not in self._tried] + if bs: + jobs.append((a, bs)) + + def run_job(job): + a, bs = job + for k, b in enumerate(bs): + T = self._match(a, b) + if T is not None: + return (a, b, T, SIGMA_LOOP), k + 1 + return None, len(bs) + new = 0 + for (a, bs), (edge, tries) in zip(jobs, self._pool.map(run_job, jobs)): + self._tried.update((a, b) for b in bs[:tries]) + self.stats["loop_tries"] += tries + if edge is not None: + self.loops.append(edge) + new += 1 + return new + + def solve(self, rounds=4): + """Poses (N, 3) after loop closing and optimisation.""" + self._feat, self._tried, self.loops = {}, set(), [] + self.stats.update(loop_tries=0, loops=0, pruned=0) + x = self._chain() + keys = self._keyframes(x) + self._pool = ThreadPoolExecutor(os.cpu_count()) + self._feat.update(zip(keys, self._pool.map(self._detect, keys))) + base = self.odo + self.bridges + for r in range(rounds): + # Wide search first (drift can be hundreds of px), tighter once poses are good. + new = self._close_loops(x, keys, radius=700 if r == 0 else 500) + for _ in range(5): + x, nrm = self._optimise(x, base + self.loops) + bad = nrm[len(base):] > PRUNE + if not bad.any(): + break + self.loops = [e for e, b in zip(self.loops, bad) if not b] + self.stats["pruned"] += int(bad.sum()) + if r and not new: + break + self.anchored = self._main_component() + self._verify(x) + self._pool.shutdown() + self._pool, self._feat = None, {} + self.stats["loops"] = len(self.loops) + self.stats["rendered"] = int(sum(self.anchored)) + self.residuals = nrm + return x + + def _main_component(self): + """Frames linked to the largest group by image-verified edges. Off-page stretches + register frame to frame but connect to the page only through the lasers.""" + parent = list(range(self.n)) + + def find(k): + while parent[k] != k: + parent[k] = parent[parent[k]] + k = parent[k] + return k + for i, j, _, s in self.odo + self.bridges + self.loops: + if s != SIGMA_LASER: + parent[find(i)] = find(j) + roots = np.array([find(k) for k in range(self.n)]) + main = np.bincount(roots).argmax() + return list(roots == main) + + def _verify(self, x, min_ncc=0.5, scale=0.5): + """Drop frames that disagree with the mosaic of all frames (a stretch held only by a + wrong registration). Compared on high-passed images at half resolution.""" + use = [i for i in range(self.n) if self.anchored[i]] + S = np.diag([scale, scale, 1.0]) + Ms, off, W, H = self._layout(x, use, S) + fe = cv2.resize(_feather(self.h, self.w), None, fx=scale, fy=scale) + small = list(self._pool.map( + lambda i: cv2.resize(self.hp[i], None, fx=scale, fy=scale, interpolation=cv2.INTER_AREA), use)) + A = [off @ M @ np.linalg.inv(S) for M in Ms] + acc = np.zeros((H, W), np.float32) + ws = np.zeros((H, W), np.float32) + for im, M in zip(small, A): + _warp_add(acc, im * fe, M) + _warp_add(ws, fe, M) + mos = acc / np.maximum(ws, 1e-6) + h, w = fe.shape + + def score(k): + back = cv2.warpAffine(mos, np.linalg.inv(A[k])[:2], (w, h)) + return np.corrcoef(small[k][10:-10, 10:-10].ravel(), back[10:-10, 10:-10].ravel())[0, 1] + self.ncc = np.full(self.n, np.nan) + self.ncc[use] = list(self._pool.map(score, range(len(use)))) + for i in use: + if not self.ncc[i] >= min_ncc: + self.anchored[i] = False + self.stats["inconsistent"] = int(np.sum(self.ncc < min_ncc)) + + def _layout(self, x, use, S=np.eye(3)): + """Poses of frames `use` scaled by S, the offset that puts them all on a canvas, and its size.""" + Ms = [S @ _aff(x[i, 2], x[i, 0], x[i, 1]) for i in use] + c = np.array([[0, 0, 1], [self.w, 0, 1], [0, self.h, 1], [self.w, self.h, 1.0]]).T + q = np.concatenate([(M @ c)[:2] for M in Ms], 1) + off = _aff(0, 2 - q[0].min(), 2 - q[1].min()) + W, H = int(np.ceil(np.ptp(q[0]))) + 4, int(np.ceil(np.ptp(q[1]))) + 4 + return Ms, off, W, H + + # --- rendering --------------------------------------------------------------- + + def image(self, x=None, white_balance=True, background=255): + x = self.solve() if x is None else x + use = [i for i in range(self.n) if self.anchored[i]] + Ms, off, W, H = self._layout(x, use) + ch = self.frames[0].shape[2] if self.frames[0].ndim == 3 else 0 + acc = np.zeros((H, W, ch) if ch else (H, W), np.float32) + wsum = np.zeros((H, W), np.float32) + feather = _feather(self.h, self.w) + for i, M in zip(use, Ms): + f = self._prep(self.frames[i]) + _warp_add(acc, f * (feather[..., None] if f.ndim == 3 else feather), off @ M) + _warp_add(wsum, feather, off @ M) + return compose(acc, wsum, white_balance, background) diff --git a/myscan/stitch.py b/myscan/stitch.py index 866767c..7e75a29 100644 --- a/myscan/stitch.py +++ b/myscan/stitch.py @@ -177,18 +177,23 @@ class Stitcher: self.n += 1 def image(self, white_balance=True, background=255): - covered = self.wsum > 0.05 - ws = np.maximum(self.wsum, 1e-6) - out = self.acc / (ws[..., None] if self.ch > 1 else ws) - if white_balance: - out = white_patch(out, covered) - m = covered[..., None] if self.ch > 1 else covered - out = np.where(m, out, background).clip(0, 255).astype(np.uint8) - ys, xs = np.nonzero(covered) - if len(ys): - out = out[ys.min():ys.max() + 1, xs.min():xs.max() + 1] - # Frames arrive transposed and mirrored relative to the page. - return cv2.flip(cv2.transpose(out), -1) + return compose(self.acc, self.wsum, white_balance, background) + + +def compose(acc, wsum, white_balance=True, background=255): + """Weighted canvas (acc, wsum) -> cropped page image in page orientation.""" + covered = wsum > 0.05 + ws = np.maximum(wsum, 1e-6) + out = acc / (ws[..., None] if acc.ndim == 3 else ws) + if white_balance: + out = white_patch(out, covered) + m = covered[..., None] if acc.ndim == 3 else covered + out = np.where(m, out, background).clip(0, 255).astype(np.uint8) + ys, xs = np.nonzero(covered) + if len(ys): + out = out[ys.min():ys.max() + 1, xs.min():xs.max() + 1] + # Frames arrive transposed and mirrored relative to the page. + return cv2.flip(cv2.transpose(out), -1) def white_patch(img, mask, pct=97, white=245): diff --git a/requirements.txt b/requirements.txt index 959d583..1fb92b7 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,3 @@ numpy opencv-python-headless +scipy diff --git a/tools/scan.py b/tools/scan.py index d845b25..5077b2f 100644 --- a/tools/scan.py +++ b/tools/scan.py @@ -1,12 +1,15 @@ #!/usr/bin/env python3 """Scan with the scan button, like the original app: press it to start, glide the -mouse over the page, press it again to stop. The page is stitched live and saved -as /scan-YYYYmmdd-HHMMSS.png; the mouse goes back to mouse mode between -scans (and on Ctrl-C).""" +mouse over the page, press it again to stop. Frames are registered while you scan; +after the stop press the page is assembled from all of them at once (see +myscan/pagegraph.py) and saved as /scan-YYYYmmdd-HHMMSS.png. The mouse goes +back to mouse mode between scans (and on Ctrl-C).""" import argparse import csv import os +import queue import sys +import threading import time sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) @@ -17,7 +20,7 @@ quiet_libjpeg() import cv2 # noqa: E402 from myscan import calib # noqa: E402 -from myscan.stitch import Stitcher # noqa: E402 +from myscan.pagegraph import PageGraph # noqa: E402 from myscan.xu import XU # noqa: E402 @@ -34,16 +37,27 @@ def wait_press(x, poll=0.05): time.sleep(poll) -def scan_page(x, a, frames_dir=None, flat=None, gains=None): - st, rows = None, [] +def scan_page(x, a, frames_dir=None, flat=None, undistort=None, gains=None): + """Capture until the button is pressed again. Frames are registered (and saved) on a + worker thread so capture never waits; returns the PageGraph, or None.""" + g, rows, work = None, [], queue.Queue() + + def worker(): + while (item := work.get()) is not None: + n, im, v = item + g.add(im, v) + if frames_dir: + cv2.imwrite(os.path.join(frames_dir, f"f{n:05d}.png"), im) + th = None with ScanSession(a.video, a.target, a.exposure, xu=x, gains=gains) as s: down, last = 0, 0.0 for t, n, im, rec in s.frames(color=not a.gray): - if st is None: - st = Stitcher(im.shape, refine=not a.no_refine, flat=flat) - st.add(im, rec.v) + if g is None: + g = PageGraph(im.shape, flat=flat, undistort=undistort) + th = threading.Thread(target=worker, daemon=True) + th.start() + work.put((n, im, rec.v)) if frames_dir: - cv2.imwrite(os.path.join(frames_dir, f"f{n:05d}.png"), im) rows.append([round(t, 4), n, rec.counter, *rec.v, rec.b14, rec.b15, rec.b16, rec.flags, s.e6, round(float(im.mean()), 1), rec.raw.hex()]) # Stop on a press: two "down" reads in a row (debounce), ignoring the first @@ -56,17 +70,19 @@ def scan_page(x, a, frames_dir=None, flat=None, gains=None): print("max scan time reached", flush=True) break if t - last >= 1: - print(f"\r {t:5.1f}s {st.n} frames canvas {st.size[0]}x{st.size[1]} E6={s.e6}", - end="", flush=True) + print(f"\r {t:5.1f}s {n + 1} frames ({g.n} registered) E6={s.e6}", end="", flush=True) last = t print() + if th: + work.put(None) + th.join() if frames_dir: # same layout as tools/record.py, so tools/stitch.py can replay it with open(os.path.join(frames_dir, "records.csv"), "w", newline="") as fh: w = csv.writer(fh) w.writerow(["t", "frame", "counter", "v0", "v1", "v2", "v3", "v4", "v5", "b14", "b15", "b16", "flags", "e6", "brightness", "raw"]) w.writerows(rows) - return st + return g def main(): @@ -80,7 +96,7 @@ def main(): ap.add_argument("--gray", action="store_true") ap.add_argument("--no-wb", action="store_true", help="skip white-patch balancing") ap.add_argument("--no-flat", action="store_true", help="skip the EEPROM flat-field correction") - ap.add_argument("--no-refine", action="store_true", help="lasers only, no image registration") + ap.add_argument("--no-undistort", action="store_true", help="skip the EEPROM lens correction") ap.add_argument("--save-frames", action="store_true", help="also keep raw frames + records.csv per scan") ap.add_argument("--max-time", type=float, default=180, help="stop a scan after this many seconds") a = ap.parse_args() @@ -89,6 +105,7 @@ def main(): print("reading calibration from the mouse's EEPROM (cached after the first time)", flush=True) cal = calib.load(x) flat = None if a.no_flat else calib.FlatField(cal) + undistort = None if a.no_undistort else calib.Undistort(cal) gains = cal["gains_rgb"] first = True try: @@ -102,13 +119,16 @@ def main(): frames_dir = os.path.join(a.outdir, name) if a.save_frames else None if frames_dir: os.makedirs(frames_dir) - st = scan_page(x, a, frames_dir, flat, gains) - if st is None or st.n < 2: + g = scan_page(x, a, frames_dir, flat, undistort, gains) + if g is None or g.n < 2: print("no frames captured") else: + print("assembling the page...", flush=True) + t0 = time.time() + page = g.image(white_balance=not a.no_wb) path = os.path.join(a.outdir, name + ".png") - cv2.imwrite(path, st.image(white_balance=not a.no_wb)) - print(f"saved {path} ({st.n} frames, {st.stats})", flush=True) + cv2.imwrite(path, page) + print(f"saved {path} ({time.time() - t0:.1f} s; {g.stats})", flush=True) if a.once: break except KeyboardInterrupt: diff --git a/tools/stitch.py b/tools/stitch.py index b3d3534..3af9193 100644 --- a/tools/stitch.py +++ b/tools/stitch.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Stitch a directory recorded by tools/record.py into a page image.""" +"""Stitch a directory recorded by tools/record.py (or scan.py --save-frames) into a page image.""" import argparse import csv import os @@ -11,6 +11,7 @@ import cv2 # noqa: E402 import numpy as np # noqa: E402 from myscan import calib # noqa: E402 +from myscan.pagegraph import PageGraph # noqa: E402 from myscan.stitch import ROT_GAIN, Stitcher # noqa: E402 @@ -18,9 +19,11 @@ def main(): ap = argparse.ArgumentParser() ap.add_argument("dir") ap.add_argument("-o", "--out", help="output image (default /page.png)") - ap.add_argument("--no-refine", action="store_true", help="lasers only, no image registration") + ap.add_argument("--live", action="store_true", help="use the live frame-by-frame stitcher instead of the global one") + ap.add_argument("--no-refine", action="store_true", help="with --live: lasers only, no image registration") ap.add_argument("--rot-gain", type=float, default=ROT_GAIN) ap.add_argument("--no-flat", action="store_true", help="skip the EEPROM flat-field correction") + ap.add_argument("--no-undistort", action="store_true", help="skip the EEPROM lens correction") ap.add_argument("--no-wb", action="store_true", help="skip white-patch balancing") ap.add_argument("--start", type=int, default=0) ap.add_argument("--end", type=int) @@ -31,12 +34,15 @@ def main(): frames = [int(r["frame"]) for r in rows] rng = range(a.start, a.end if a.end is not None else len(rows)) - flat = None if a.no_flat else calib.FlatField(calib.load()) + cal = calib.load() + flat = None if a.no_flat else calib.FlatField(cal) + undistort = None if a.no_undistort else calib.Undistort(cal) st = None for i in rng: im = cv2.imread(os.path.join(a.dir, f"f{frames[i]:05d}.png"), cv2.IMREAD_UNCHANGED) if st is None: - st = Stitcher(im.shape, refine=not a.no_refine, rot_gain=a.rot_gain, flat=flat) + st = (Stitcher(im.shape, refine=not a.no_refine, rot_gain=a.rot_gain, flat=flat) if a.live + else PageGraph(im.shape, flat=flat, undistort=undistort, rot_gain=a.rot_gain)) st.add(im, v[i]) out = a.out or os.path.join(a.dir, "page.png") cv2.imwrite(out, st.image(white_balance=not a.no_wb))