Compare commits

...
10 Commits
Author SHA1 Message Date
erikandClaude Opus 5.5 04c10e58c7 Drop frames taken while the mouse lies still
PageGraph.add skips a frame when neither the laser odometry nor a small
thumbnail shows it moved since the last kept frame, so resting on the page
no longer costs registration, solving and rendering time. LiveSession maps
captures to the kept frames for its sensor readings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:22:27 +00:00
erikandClaude Opus 5.5 eeea9c3b30 Add an optional Enhance filter that removes bleed-through
Enhance (Ctrl+E) flattens the paper to white, darkens the ink and
whitens marks fainter than the ink, such as print showing through from
the other side. Faint pixels next to ink keep their shading, so strokes
keep their edges. Coloured ink keeps its hue. A dropdown sets the
strength (Light, Normal, Strong). OCR and Save use the page as shown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:13:44 +00:00
erikandClaude Opus 5.5 5f9ecca302 Make only a scan's unscanned area transparent, not every white pixel
The page view made every pure-white pixel transparent. The white point
puts the paper's 97th percentile at 245, so the brightest paper clips
to 255 and showed as dark speckles; opened images with a white
background turned dark entirely; and the live preview (unscanned area
already dark) only got the speckles. Now only fresh scans are masked,
and only the white regions connected to the border, which is where the
stitcher's uncovered area is after cropping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:55:29 +00:00
erikandClaude Opus 4.7 e9ab616161 Drop the Zoom +/− toolbar actions; show the window through the page's uncovered corners
The wheel (and Fit, Ctrl+0) remain. The stitched page has white in
areas that were never scanned; when the view rotates it to level the
text, those triangles stuck out white against the dark window. The
page is now shown with its exact-white pixels transparent, so the
window colour shows through there instead.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-10-11 21:49:02 +00:00
erikandClaude Opus 4.7 0dd807a4e7 Add a sensor dial under the camera with live scan readings
Shows the mouse's rotation on the page since the scan started (a big "A",
coloured by tracking), direction and speed (arrow), and heading, speed,
distance, offset, area, tracking, laser quality, exposure, paper level
and fps. Also drops the "Read text", "Auto" and "Save text" toolbar
actions (OCR runs automatically on every finished page or opened image).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-10-11 21:44:03 +00:00
erikandClaude Opus 5.5 61b6cf70a7 Add a Camera toggle to show the camera panel, hidden by default
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:25:26 +00:00
erikandClaude Opus 5.5 16df509d01 Level the page view by the text direction found by OCR
The page, its line boxes and the selection now hang under one rotated item, so
selection works the same whether the page is levelled or shown as stitched.
Level (Ctrl+L, on by default) toggles it, and Save image saves what is shown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:06:59 +00:00
erikandClaude Opus 5.5 2ef567580a Decide text orientation per page instead of per line
RapidOCR's angle classifier judges each cut-out line on its own and now
and then turns a perfectly upright line upside down, which then reads as
garbage ("Jo a ds ne t i d s t d no" for a line ending in "or" on the
latest scan). It is off now. All lines on a page face the same way, so a
page that reads poorly (upright text scores 0.97-1.0 per character,
upside-down text 0.55-0.8) is read again rotated 180 degrees and the
better reading kept; that also covers pages turned 90 degrees. The app
shows an upside-down page turned the right way up.

Upright pages read about 25% faster without the per-line classifier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 12:44:40 +00:00
erikandClaude Opus 5.5 a1942faa41 Order OCR lines in the page's own frame, so slanted scans read right
Lines were grouped into rows and sorted by their centres' image y and
their left x. On a page scanned at an angle (5.4 degrees on the latest
scan) a short line's centre sits lower than that of a long line beside
it, so lines got swapped or merged into one row and paragraphs read as
nonsense. Rows now use positions along and across the text direction,
the length-weighted median direction of the lines (ocr.text_direction,
ocr.page_coords); a rotated page gives the same text.

Paragraph joining also follows a list item's hanging indent (continuation
lines lined up after "(a)", "1.", a bullet ...) and lets a heading whose
number was read as a separate item run on to its next line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 12:44:40 +00:00
erikandClaude Opus 5.5 6631073be5 Copy OCR text as paragraphs
A Paragraphs toggle (Ctrl+J, on by default) makes copied and saved text
put each paragraph on one line: line breaks inside a paragraph become
spaces and words hyphenated across lines are rejoined (the hyphen stays
before a capital, as in "Nord-Amerika"). A line ends its paragraph if the
next one is indented, has a different text size, or starts with a word
that would have fitted at the end of this one, which works for justified
and ragged-right text alike; rows of several items (tables) are left
alone. The right-click menu offers the other form.

ocr.paragraph_joins gives the joins as edits on layout()'s text, and
reflow applies those inside any range, so selections keep working on the
page's character positions.

Ctrl+C now copies in the chosen form (QAction.triggered passed
checked=False through as paragraphs=False), and tabs show in the status
bar's selection preview.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 12:44:40 +00:00
7 changed files with 687 additions and 102 deletions
+14 -4
View File
@@ -13,18 +13,28 @@ Linux driver tools for the Hama myScan (00052343) scanner mouse, a Dacuda design
prior; another device can pass its own object with `predict()`), `register` (ECC, phase correlation, SIFT
matching, overlap check), `canvas` (weighted canvases, page composition, parallel rendering), `posegraph`
(robust pose-graph optimisation), `loops` (loop-closure candidates), and the two stitchers built from them:
`pagegraph` (`PageGraph`: assembles the page from all frames at once with loop closures between passes;
`pagegraph` (`PageGraph`: assembles the page from all frames at once with loop closures between passes,
dropping frames taken while the mouse lies still;
used by `scan.py` and `stitch.py`) and `incremental` (`Stitcher`: frame by frame, `stitch.py --live`)
- `tools/scan.py` — scan with the button: `python tools/scan.py` (press to start, press again to stop).
`--view [PORT]` serves a live view at `http://<host>:8000/`: the camera, the page as it is stitched
(re-solved every 1.5 s, current position outlined), Start/Stop buttons, PNG/JPEG download of the finished
page and a Screenshot button (camera + page + status as one PNG)
- `tools/app.py` — desktop app (Qt/PySide6; Linux, Windows, macOS): live camera and page view while scanning
(Start/Stop or the scan button, F5), then OCR of the finished page with RapidOCR (`myscan/ocr.py`;
multilingual PP-OCRv6 models bundled in the wheel, works offline). Recognised lines are outlined on the page by
(Start/Stop or the scan button, F5). Under the camera a dial shows how the mouse is turned on the page since the
scan started (a big "A", coloured by tracking), its direction and speed (arrow), and the live readings: speed,
distance, offset from the start, area covered, tracking (share of recent frames matched by image rather than
the lasers alone), the lasers' surface quality, exposure, paper brightness and frame rate. Then OCR of the finished page with RapidOCR (`myscan/ocr.py`;
multilingual PP-OCRv6 models bundled in the wheel, works offline; an upside-down page is detected and turned). Level (Ctrl+L, on by default) shows the page rotated so its lines
are horizontal, using the text direction found by OCR; switch it off to see the page as stitched. Save image saves
what is shown. Recognised lines are outlined on the page by
confidence, and the text is selected right on the page like in a PDF viewer: drag (in reading order across
lines), double-click a word, triple-click a line, shift-click to extend, Ctrl+A; Ctrl+C or right-click copies.
Middle-drag or Space+drag pans, the wheel zooms. Copy all and Save text (.txt) take the whole page. `python tools/app.py [image]` also opens and reads existing images (Open, or
With Paragraphs on (Ctrl+J, the default), copied text has each paragraph on one line:
line breaks inside a paragraph become spaces and words hyphenated across lines are rejoined; a line ends a
paragraph if the next line is indented or its first word would have fitted on it (works for justified and
ragged text), and rows with several columns are left alone. Right-click offers the other form. Middle-drag or
Space+drag pans, the wheel zooms. Copy all takes the whole page. `python tools/app.py [image]` also opens and reads existing images (Open, or
drag and drop), which is all it does on machines without the mouse
- `myscan/scanner.py` — the scan loop shared by `scan.py` and the app (finds the mouse's `/dev/videoN` itself)
- `myscan/live.py`, `myscan/webview.py` — the live preview and its browser viewer (standard library only)
+333 -69
View File
@@ -4,7 +4,8 @@ Left, the camera and the page as it is being stitched (the same preview as the b
viewer); after a scan the finished page is loaded and read with RapidOCR (myscan/ocr.py).
Recognised lines are outlined on the page, coloured by confidence, and their text can be
selected right on the page (drag, double-click a word, triple-click a line) and copied;
all of it can be copied or saved. Images can also be opened or dropped onto the window,
all of it can be copied or saved. Enhance whitens the paper, darkens the text and drops
bleed-through from the other side (myscan/enhance.py); OCR and Save use what is shown. Images can also be opened or dropped onto the window,
so OCR works on any platform; scanning needs Linux and the mouse.
Run: python tools/app.py [image]
@@ -19,23 +20,43 @@ import traceback
import cv2
import numpy as np
from PySide6.QtCore import QObject, QPointF, QRectF, Qt, QThread, QTimer, Signal
from PySide6.QtGui import (QAction, QBrush, QClipboard, QColor, QGuiApplication, QImage, QKeySequence, QPainter,
QPainterPath, QPen, QPixmap, QPolygonF)
from PySide6.QtGui import (QAction, QActionGroup, QBrush, QClipboard, QColor, QFont, QGuiApplication, QImage, QKeySequence,
QPainter, QPainterPath, QPen, QPixmap, QPolygonF)
from PySide6.QtWidgets import (QApplication, QFileDialog, QGraphicsPathItem, QGraphicsPixmapItem,
QGraphicsPolygonItem, QGraphicsScene, QGraphicsView, QLabel, QMainWindow, QMenu,
QMessageBox, QSizePolicy, QSplitter, QToolBar, QVBoxLayout, QWidget)
QGraphicsPolygonItem, QGraphicsRectItem, QGraphicsScene, QGraphicsView, QLabel, QMainWindow, QMenu,
QMessageBox, QSizePolicy, QSplitter, QToolBar, QToolButton, QVBoxLayout, QWidget)
from .ocr import Ocr, layout
from .enhance import enhance
from .ocr import Ocr, layout, paragraph_joins, reflow, text_direction
IMAGE_FILTER = "Images (*.png *.jpg *.jpeg *.tif *.tiff *.bmp *.webp)"
# Enhance strengths (see enhance.enhance): how much fainter than the ink a mark must be to
# be whitened as bleed-through. Stronger removes darker bleed-through and more faint lines.
ENHANCE_LEVELS = (("Light", 0.15), ("Normal", 0.25), ("Strong", 0.5))
def to_qimage(img):
"""BGR or grayscale uint8 array -> QImage (a copy)."""
def to_qimage(img, transparent=None):
"""BGR or grayscale uint8 array -> QImage (a copy); pixels where the bool mask
`transparent` is set get alpha 0."""
img = np.ascontiguousarray(img)
h, w = img.shape[:2]
fmt = QImage.Format_Grayscale8 if img.ndim == 2 else QImage.Format_BGR888
return QImage(img.data, w, h, img.strides[0], fmt).copy()
if transparent is None:
fmt = QImage.Format_Grayscale8 if img.ndim == 2 else QImage.Format_BGR888
return QImage(img.data, w, h, img.strides[0], fmt).copy()
bgra = cv2.cvtColor(img, cv2.COLOR_GRAY2BGRA if img.ndim == 2 else cv2.COLOR_BGR2BGRA)
bgra[transparent, 3] = 0
return QImage(bgra.data, w, h, bgra.strides[0], QImage.Format_ARGB32).copy()
def unscanned(img):
"""Mask of a stitched page's unscanned area: the pure-white regions touching the
border (the stitcher fills uncovered pixels with 255 and crops to the scanned area).
Paper that clips to 255 inside the page is not connected to the border and stays."""
white = ((img == 255).all(axis=-1) if img.ndim == 3 else img == 255).astype(np.uint8)
_, labels = cv2.connectedComponents(white, connectivity=4)
edge = np.concatenate([labels[0], labels[-1], labels[:, 0], labels[:, -1]])
edge = np.unique(edge[edge > 0])
return np.isin(labels, edge) if len(edge) else None
def imread(path):
@@ -50,6 +71,17 @@ def imwrite(path, img):
buf.tofile(path)
def level_image(img, angle):
"""The image rotated by -angle degrees (clockwise in image coordinates, y down), as
PageView shows it levelled, on a white canvas big enough to hold all of it."""
h, w = img.shape[:2]
m = cv2.getRotationMatrix2D((w / 2, h / 2), angle, 1.0)
c, s = abs(m[0, 0]), abs(m[0, 1])
W, H = int(np.ceil(w * c + h * s)), int(np.ceil(w * s + h * c))
m[:, 2] += ((W - w) / 2, (H - h) / 2)
return cv2.warpAffine(img, m, (W, H), flags=cv2.INTER_CUBIC, borderValue=(255, 255, 255))
def log_error(msg):
"""Print an error, with the traceback of the exception being handled if any."""
print(f"myScan: {msg}", file=sys.stderr, flush=True)
@@ -136,7 +168,7 @@ class ScanThread(QThread):
class OcrThread(QThread):
done = Signal(object, float) # lines, seconds
done = Signal(object, bool, float) # lines, page upside down, seconds
failed = Signal(str)
def __init__(self, ocr, img):
@@ -146,7 +178,7 @@ class OcrThread(QThread):
def run(self):
t = time.time()
try:
self.done.emit(self.ocr(self.img), time.time() - t)
self.done.emit(*self.ocr.read(self.img), time.time() - t)
except Exception as e:
log_error(f"OCR failed: {e}")
self.failed.emit(str(e))
@@ -169,8 +201,14 @@ class LineItem(QGraphicsPolygonItem):
class PageView(QGraphicsView):
"""Zoomable page whose recognised text can be selected like in a PDF viewer: drag to
select in reading order, double-click for a word, triple-click for a line, shift-click
to extend. Middle-drag (or Space + drag, or any drag before there is text) pans. Keeps
the page fitted to the window until the user zooms."""
to extend. Copied text either keeps the page's line breaks or, with `paragraphs` set,
has each paragraph on one line (see ocr.paragraph_joins). Middle-drag (or Space +
drag, or any drag before there is text) pans. Keeps the page fitted to the window
until the user zooms.
The page, its line boxes and the selection are children of one item, which is
rotated to level the text (set_angle, set_level), so selection works the same either
way: everything is in image coordinates under that item."""
selectionChanged = Signal(str)
def __init__(self):
@@ -180,35 +218,57 @@ class PageView(QGraphicsView):
self.setTransformationAnchor(QGraphicsView.AnchorUnderMouse)
self.setBackgroundBrush(QColor(30, 32, 36))
self.setMouseTracking(True)
self.pixmap = QGraphicsPixmapItem()
self.page = QGraphicsRectItem()
self.page.setPen(QPen(Qt.NoPen))
self.scene().addItem(self.page)
self.pixmap = QGraphicsPixmapItem(self.page)
self.pixmap.setTransformationMode(Qt.SmoothTransformation)
self.scene().addItem(self.pixmap)
self.sel_item = QGraphicsPathItem()
self.sel_item = QGraphicsPathItem(self.page)
self.sel_item.setPen(QPen(Qt.NoPen))
self.sel_item.setBrush(QColor(30, 110, 230, 90))
self.sel_item.setZValue(2)
self.scene().addItem(self.sel_item)
self.angle, self.level = 0.0, True
self.items_, self.auto_fit, self.boxes_visible = [], True, True
self.lines, self.spans, self.text = [], [], ""
self.lines, self.spans, self.text, self.joins, self.paragraphs = [], [], "", [], True
self.sel, self.anchor, self.unit = (0, 0), None, None # unit: drag extends by "word"/"line"
self.pan_from, self.space, self.last_double = None, False, None
def set_image(self, qimg, keep_view=False):
self.pixmap.setPixmap(QPixmap.fromImage(qimg))
self.scene().setSceneRect(QRectF(self.pixmap.pixmap().rect()))
if not keep_view:
self.auto_fit = True
self.place()
def set_angle(self, angle):
"""The text's slant in degrees (clockwise, as in image coordinates)."""
self.angle = angle
self.place()
def set_level(self, on):
self.level = on
self.place()
def place(self):
"""Rotate the page to level it (or not), and fit the scene around it."""
r = self.pixmap.boundingRect()
self.page.setTransformOriginPoint(r.center())
self.page.setRotation(-self.angle if self.level else 0.0)
self.scene().setSceneRect(self.page_rect())
if self.auto_fit:
self.fit()
def page_rect(self):
return self.page.mapRectToScene(self.pixmap.boundingRect())
def set_lines(self, lines, text="", spans=()):
for it in self.items_:
self.scene().removeItem(it)
self.items_ = [LineItem(ln.box, ln.score) for ln in lines]
for it in self.items_:
it.setVisible(self.boxes_visible)
self.scene().addItem(it)
it.setParentItem(self.page)
self.lines, self.text, self.spans = lines, text, list(spans)
self.joins = paragraph_joins(lines, text, self.spans) if lines else []
self.sel, self.anchor = None, None
self.select(0, 0)
@@ -221,7 +281,8 @@ class PageView(QGraphicsView):
def hit(self, scene_pos):
"""(line index, distance in pixels, text position) of the line nearest a scene point."""
p = (scene_pos.x(), scene_pos.y())
q = self.page.mapFromScene(scene_pos)
p = (q.x(), q.y())
return min(((i, *ln.locate(p)) for i, ln in enumerate(self.lines)), key=lambda h: h[1])
def position(self, scene_pos, unit=None):
@@ -253,10 +314,20 @@ class PageView(QGraphicsView):
def select_all(self):
self.select(0, len(self.text))
def selected_text(self):
a, b = self.sel
def text_range(self, a=0, b=None, paragraphs=None):
"""The text from a to b (positions in self.text), reflowed into paragraphs if
`paragraphs` (default: self.paragraphs)."""
if self.paragraphs if paragraphs is None else paragraphs:
return reflow(self.text, self.joins, a, b)
return self.text[a:b]
def selected_text(self, paragraphs=None):
return self.text_range(*self.sel, paragraphs=paragraphs)
def set_paragraphs(self, on):
self.paragraphs = on
self.selectionChanged.emit(self.selected_text())
def drag_to(self, scene_pos):
a, b = self.position(scene_pos, self.unit)
a0, b0 = self.anchor
@@ -334,22 +405,25 @@ class PageView(QGraphicsView):
if not self.lines:
return
m = QMenu(self)
m.addAction("Copy", self.copy).setEnabled(self.sel[0] != self.sel[1])
m.addAction("Copy all text", lambda: QGuiApplication.clipboard().setText(self.text))
has_sel = self.sel[0] != self.sel[1]
other = "Copy with the page's line breaks" if self.paragraphs else "Copy as paragraphs"
m.addAction("Copy", self.copy).setEnabled(has_sel)
m.addAction(other, lambda: self.copy(not self.paragraphs)).setEnabled(has_sel)
m.addAction("Copy all text", lambda: QGuiApplication.clipboard().setText(self.text_range()))
m.addSeparator()
m.addAction("Select all", self.select_all)
m.exec(ev.globalPos())
def copy(self):
if self.selected_text():
QGuiApplication.clipboard().setText(self.selected_text())
def copy(self, paragraphs=None):
if text := self.selected_text(paragraphs):
QGuiApplication.clipboard().setText(text)
# ---- zoom ----
def fit(self):
self.auto_fit = True
if not self.pixmap.pixmap().isNull():
self.fitInView(self.pixmap, Qt.KeepAspectRatio)
self.fitInView(self.page_rect(), Qt.KeepAspectRatio)
def zoom(self, f):
self.auto_fit = False
@@ -389,6 +463,102 @@ class ImageLabel(QLabel):
Qt.SmoothTransformation))
class SensorDial(QWidget):
"""Live readings while scanning (LiveSession.sensors): a dial with a big "A" turned as
the mouse is turned on the page (upright: as at the start of the scan; the page preview
is drawn in that orientation), an arrow for the direction and speed of motion, and the
other readings listed below it."""
FULL_SPEED = 50.0 # mm/s for a full-length arrow; the lasers start to miscount near this
ROWS = 10
def __init__(self):
super().__init__()
self.s = None
self.setMinimumSize(120, 160)
self.setSizePolicy(QSizePolicy.Preferred, QSizePolicy.Fixed)
self.setFixedHeight(200 + 17 * self.ROWS)
def set_sensors(self, s):
self.s = s
self.update()
def rows(self):
s = self.s
quality = lambda f: "#2ea043" if f >= 0.8 else "#db8c00" if f >= 0.5 else "#d62828"
fast = s["speed"] > 0.8 * self.FULL_SPEED
return [
("Heading", f"{s['heading']:+.1f}°", None),
("Speed", f"{s['speed']:.0f} mm/s", "#db8c00" if fast else None),
("Distance", f"{s['distance'] / 10:.1f} cm", None),
("Offset", "{:+.0f}, {:+.0f} mm".format(*s["offset"]), None),
("Area", f"{s['area']:.1f} cm²", None),
("Tracking", f"{s['tracking']:.0%}", quality(s["tracking"])),
("Lasers", "{} / {}".format(*s["lasers"]), None),
("Exposure", f"{s['exposure']:.2f} ms", None),
("Paper level", f"{s['level']:.0f} / 255", None),
("Frames", f"{s['fps']:.0f}/s", None),
]
def paintEvent(self, ev):
p = QPainter(self)
p.setRenderHint(QPainter.Antialiasing)
p.fillRect(self.rect(), QColor(30, 32, 36))
grey, light = QColor(139, 145, 154), QColor(220, 222, 226)
s = self.s
r = min(self.width() - 16, 184) / 2
cx, cy = self.width() / 2, 8 + r
p.save()
p.translate(cx, cy)
p.setPen(QPen(QColor(70, 74, 82), 1.5))
p.setBrush(QColor(24, 26, 30))
p.drawEllipse(QPointF(0, 0), r, r)
for deg in range(0, 360, 15): # ticks; long ones every 90°, page up at the top
p.save()
p.rotate(deg)
p.setPen(QPen(light if deg == 0 else grey, 2 if deg % 90 == 0 else 1))
p.drawLine(QPointF(0, -r), QPointF(0, -r + (9 if deg % 90 == 0 else 5)))
p.restore()
if s:
p.save()
p.rotate(s["heading"])
f = QFont(self.font())
f.setPixelSize(int(r * 1.2))
f.setBold(True)
glyph = QPainterPath()
glyph.addText(0, 0, f, "A")
glyph.translate(-glyph.boundingRect().center()) # turn about the letter's own centre
colour = QColor("#2ea043" if s["tracking"] >= 0.8 else "#db8c00" if s["tracking"] >= 0.5 else "#d62828")
p.setPen(Qt.NoPen)
p.setBrush(colour)
p.drawPath(glyph)
p.restore()
vx, vy = s["velocity"]
k = min(1.0, s["speed"] / self.FULL_SPEED) * (r - 6) / max(s["speed"], 1e-9)
if s["speed"] >= 1:
tip = QPointF(vx * k, vy * k)
p.setPen(QPen(QColor(90, 170, 255), 3, Qt.SolidLine, Qt.RoundCap))
p.drawLine(QPointF(0, 0), tip)
a = np.arctan2(vy, vx)
head = [tip + QPointF(10 * np.cos(a + d), 10 * np.sin(a + d)) for d in (np.pi * 0.85, -np.pi * 0.85)]
p.setBrush(QColor(90, 170, 255))
p.drawPolygon(QPolygonF([tip, *head]))
else:
p.setPen(grey)
p.drawText(QRectF(-r, -r, 2 * r, 2 * r), Qt.AlignCenter, "scan to see\nthe mouse's angle")
p.restore()
y = 16 + 2 * r
f = QFont(self.font())
f.setPixelSize(12)
p.setFont(f)
for name, value, colour in (self.rows() if s else ()):
p.setPen(grey)
p.drawText(QRectF(8, y, self.width() - 16, 17), Qt.AlignLeft | Qt.AlignVCenter, name)
p.setPen(QColor(colour) if colour else light)
p.drawText(QRectF(8, y, self.width() - 16, 17), Qt.AlignRight | Qt.AlignVCenter, value)
y += 17
p.end()
def pane(title, widget):
w = QWidget()
lay = QVBoxLayout(w)
@@ -410,8 +580,12 @@ class MainWindow(QMainWindow):
self.opts = opts
self.ocr = Ocr()
self.image, self.path, self.lines, self.text = None, None, [], ""
self.blank = None # unscanned-area mask of the shown page (set_page)
self.enhanced = None # self.image through enhance(), made when first shown
self.enhance_strength = dict(ENHANCE_LEVELS)["Normal"]
self.page_title = "Page"
self.ocr_thread, self.scan_thread, self.scanning = None, None, False
self.page_id = 0 # bumped for every new page, so a slow OCR of an old one is not applied
self.bridge = Bridge()
self.bridge.frame.connect(self.on_frame)
self.bridge.page.connect(self.on_preview)
@@ -422,7 +596,6 @@ class MainWindow(QMainWindow):
if self.device is None:
self.act_scan.setEnabled(False)
self.act_scan.setToolTip("No myScan found (scanning needs Linux and the mouse plugged in)")
self.camera_pane.hide()
self.set_message("No scanner found: open or drop an image to read it")
else:
self.start_scanner()
@@ -446,40 +619,72 @@ class MainWindow(QMainWindow):
tb.addAction(a)
return a
self.act_camera = act("Camera", lambda on: self.camera_pane.setVisible(on), "Ctrl+K",
"Show the camera and the live sensor readings", True)
tb.addSeparator()
self.act_scan = act("Start scan", self.toggle_scan, "F5", "Start or stop a scan (or press the mouse's scan button)")
tb.addSeparator()
act("Open…", self.open_image, QKeySequence.Open, "Open an image to read")
self.act_save = act("Save image…", self.save_image, QKeySequence.Save, "Save the page image")
tb.addSeparator()
self.act_ocr = act("Read text", self.run_ocr, "Ctrl+R", "Run OCR on the page")
self.act_auto = act("Auto", lambda: None, tip="Read the text after every scan", checkable=True)
self.act_auto.setChecked(True)
self.act_copy = act("Copy all", self.copy_text, "Ctrl+Shift+C", "Copy all text to the clipboard")
self.act_savetxt = act("Save text…", self.save_text, "Ctrl+Shift+S", "Save the text as a .txt file")
self.act_para = act("Paragraphs", self.view_paragraphs, "Ctrl+J",
"Copy each paragraph as one line, undoing the page's line breaks and "
"hyphenation (off: keep the lines as on the page)", True)
self.act_para.setChecked(True)
tb.addSeparator()
self.act_level = act("Level", self.set_level, "Ctrl+L",
"Show the page rotated so its lines are horizontal (off: as stitched); "
"Save image saves what is shown", True)
self.act_level.setChecked(True)
self.act_enhance = act("Enhance", self.set_enhance, "Ctrl+E",
"Whiten the paper, darken the text and remove bleed-through from the "
"other side of the page; the text is read again from what is shown. The arrow "
"sets how strongly: stronger removes darker bleed-through, but also faint lines",
True)
levels = QMenu(self)
group = QActionGroup(self)
for name, strength in ENHANCE_LEVELS:
a = levels.addAction(name, lambda strength=strength: self.set_enhance_strength(strength))
a.setCheckable(True)
a.setChecked(strength == self.enhance_strength)
group.addAction(a)
button = tb.widgetForAction(self.act_enhance)
button.setMenu(levels)
button.setPopupMode(QToolButton.MenuButtonPopup)
self.act_boxes = act("Boxes", lambda on: self.view.show_boxes(on), "Ctrl+B", "Show recognised lines on the page", True)
self.act_boxes.setChecked(True)
act("Fit", lambda: self.view.fit(), "Ctrl+0", "Fit the page to the window")
act("Zoom +", lambda: self.view.zoom(1.25), QKeySequence.ZoomIn)
act("Zoom −", lambda: self.view.zoom(0.8), QKeySequence.ZoomOut)
for a in (self.act_save, self.act_ocr, self.act_copy, self.act_savetxt):
for a in (self.act_save, self.act_copy):
a.setEnabled(False)
self.camera = ImageLabel("camera")
self.dial = SensorDial()
self.dial.setToolTip("The mouse's rotation on the page since the scan started (the A), its direction "
"and speed (the arrow), and other live readings. Tracking is the share of recent "
"frames matched by image; the rest relied on the lasers alone (too fast, or lifted).")
self.view = PageView()
self.view.selectionChanged.connect(self.on_selection)
for key, slot in ((QKeySequence.Copy, self.view.copy), (QKeySequence.SelectAll, self.view.select_all)):
for key, slot in ((QKeySequence.Copy, lambda: self.view.copy()),
(QKeySequence.SelectAll, self.view.select_all)):
a = QAction(self)
a.setShortcut(key)
a.triggered.connect(slot)
self.addAction(a)
self.camera_pane, _ = pane("Camera", self.camera)
camera = QWidget()
lay = QVBoxLayout(camera)
lay.setContentsMargins(0, 0, 0, 0)
lay.setSpacing(4)
lay.addWidget(self.camera, 1)
lay.addWidget(self.dial)
self.camera_pane, _ = pane("Camera", camera)
page_pane, self.page_head = pane("Page", self.view)
main = QSplitter(Qt.Horizontal)
for w in (self.camera_pane, page_pane):
main.addWidget(w)
main.setSizes([220, 1180])
self.camera_pane.hide()
self.setCentralWidget(main)
self.status_label = QLabel()
self.statusBar().addWidget(self.status_label, 1)
@@ -517,13 +722,18 @@ class MainWindow(QMainWindow):
if state == "scanning" and not self.scanning: # a new scan: clear the old page
self.scanning = True
self.view.set_lines([])
self.view.angle = 0.0
self.view.set_image(QImage())
self.lines, self.text, self.image, self.path = [], "", None, None
for a in (self.act_save, self.act_ocr, self.act_copy, self.act_savetxt):
self.enhanced = None
self.page_id += 1
for a in (self.act_save, self.act_copy):
a.setEnabled(False)
self.page_head.setText("Page (scanning)")
elif state in ("idle", "done"):
self.scanning = False
if "sensors" in s and (s["sensors"] or state == "scanning"):
self.dial.set_sensors(s["sensors"])
self.act_scan.setText("Stop scan" if state == "scanning" else "Start scan")
self.act_scan.setEnabled(state in ("idle", "scanning", "done"))
parts = [state]
@@ -542,11 +752,11 @@ class MainWindow(QMainWindow):
def on_result(self, path):
self.scanning = False
self.load(path, auto_ocr=self.act_auto.isChecked())
self.load(path, scanned=True)
# ---- images ----
def load(self, path, auto_ocr=True):
def load(self, path, scanned=False):
try:
img = imread(path)
except OSError:
@@ -555,20 +765,51 @@ class MainWindow(QMainWindow):
log_error(f"cannot read {path}")
QMessageBox.warning(self, "myScan", f"cannot read {path}")
return
self.set_page(img, path)
if auto_ocr:
self.run_ocr()
self.set_page(img, path, scanned)
self.run_ocr()
def set_page(self, img, path=None):
def set_page(self, img, path=None, scanned=False):
"""`scanned`: a fresh stitched page, whose unscanned corners are shown transparent
(opened images are shown as they are: their white may be paper)."""
self.image, self.path, self.lines, self.text = img, path, [], ""
self.blank = unscanned(img) if scanned else None
self.enhanced = None
self.page_id += 1
self.view.set_lines([])
self.view.set_image(to_qimage(img))
self.view.angle = 0.0
self.view.set_image(to_qimage(self.shown(), self.blank))
name = os.path.basename(path) if path else "page"
self.page_title = f"Page — {name} ({img.shape[1]}×{img.shape[0]})"
self.page_head.setText(self.page_title)
self.setWindowTitle(f"myScan — {name}")
self.act_save.setEnabled(True)
self.act_ocr.setEnabled(True)
def shown(self):
"""The page as shown, saved and read: enhanced or not."""
if not self.act_enhance.isChecked() or self.image is None:
return self.image
if self.enhanced is None:
QApplication.setOverrideCursor(Qt.WaitCursor)
try:
self.enhanced = enhance(self.image, self.blank, self.enhance_strength)
finally:
QApplication.restoreOverrideCursor()
return self.enhanced
def set_enhance(self, on):
if self.image is None:
return
self.page_id += 1 # read the page again as now shown; drop a reading under way
self.view.set_image(to_qimage(self.shown(), self.blank), keep_view=True)
self.set_message("page enhanced" if on else "page as scanned")
self.run_ocr()
def set_enhance_strength(self, strength):
self.enhance_strength, self.enhanced = strength, None
if self.act_enhance.isChecked():
self.set_enhance(True)
else:
self.act_enhance.trigger() # picking a strength turns Enhance on
def open_image(self):
path, _ = QFileDialog.getOpenFileName(self, "Open image", self.default_dir(), IMAGE_FILTER)
@@ -582,7 +823,10 @@ class MainWindow(QMainWindow):
path, _ = QFileDialog.getSaveFileName(self, "Save image", base + ".png", IMAGE_FILTER)
if path:
try:
imwrite(path, self.image)
img = self.shown()
if self.view.level and abs(self.view.angle) > 0.05:
img = level_image(img, self.view.angle)
imwrite(path, img)
except OSError as e:
log_error(f"cannot save {path}: {e}")
QMessageBox.warning(self, "myScan", f"cannot save {path}: {e}")
@@ -609,28 +853,51 @@ class MainWindow(QMainWindow):
def run_ocr(self):
if self.image is None or (self.ocr_thread and self.ocr_thread.isRunning()):
return
self.act_ocr.setEnabled(False)
self.page_head.setText(f"{self.page_title} · reading…")
self.set_message("reading text…")
self.ocr_thread = OcrThread(self.ocr, self.image)
self.ocr_thread = OcrThread(self.ocr, self.shown())
self.ocr_thread.page_id = self.page_id
self.ocr_thread.done.connect(self.on_ocr)
self.ocr_thread.failed.connect(self.on_ocr_failed)
self.ocr_thread.finished.connect(self.on_ocr_finished)
self.ocr_thread.start()
def on_ocr(self, lines, secs):
self.act_ocr.setEnabled(self.image is not None)
def on_ocr_finished(self):
if self.sender().page_id != self.page_id: # a new page came in while reading the old one
self.run_ocr()
def on_ocr(self, lines, flipped, secs):
if self.sender().page_id != self.page_id:
return
if flipped: # the boxes are on the page turned the right way up
self.image = np.ascontiguousarray(self.image[::-1, ::-1])
if self.enhanced is not None:
self.enhanced = np.ascontiguousarray(self.enhanced[::-1, ::-1])
if self.blank is not None:
self.blank = self.blank[::-1, ::-1]
self.view.set_image(to_qimage(self.shown(), self.blank))
self.lines = lines
self.text, spans = layout(lines)
self.view.set_lines(lines, self.text, spans)
d = text_direction(lines)
self.view.set_angle(float(np.degrees(np.arctan2(d[1], d[0]))) if lines else 0.0)
conf = f", mean confidence {np.mean([ln.score for ln in lines]):.0%}" if lines else ""
self.page_head.setText(f"{self.page_title} · {len(lines)} lines{conf}")
self.set_message(f"read {len(lines)} lines in {secs:.1f} s" + (" · drag over the text to select it"
if lines else ""))
notes = ["page was upside down, turned"] if flipped else []
notes += [f"text slanted {self.view.angle:+.1f}°"] if abs(self.view.angle) >= 0.1 else []
notes += ["drag over the text to select it"] if lines else []
self.set_message(" · ".join([f"read {len(lines)} lines in {secs:.1f} s", *notes]))
self.act_copy.setEnabled(bool(self.text))
self.act_savetxt.setEnabled(bool(self.text))
def set_level(self, on):
self.view.set_level(on)
if self.lines:
what = f"levelled ({-self.view.angle:+.1f}°)" if on else "as stitched"
self.set_message(f"showing the page {what}")
def on_ocr_failed(self, msg):
self.act_ocr.setEnabled(self.image is not None)
if self.sender().page_id != self.page_id:
return
self.page_head.setText(self.page_title)
self.set_message(f"OCR failed: {msg}")
@@ -640,21 +907,18 @@ class MainWindow(QMainWindow):
cb = QGuiApplication.clipboard()
if cb.supportsSelection(): # X11/Wayland: middle-click pastes it elsewhere
cb.setText(text, QClipboard.Selection)
preview = " ⏎ ".join(t for t in text.split("\n") if t)
preview = " ⏎ ".join(t for t in text.replace("\t", " ").split("\n") if t)
preview = preview if len(preview) <= 90 else preview[:87] + "…"
self.set_message(f"{len(text)} characters selected (Ctrl+C to copy): {preview}")
def copy_text(self):
QGuiApplication.clipboard().setText(self.text)
self.set_message("text copied to the clipboard")
def view_paragraphs(self, on):
self.view.set_paragraphs(on)
if not self.view.selected_text():
self.set_message("copying paragraphs as single lines" if on else "copying lines as on the page")
def save_text(self):
base = os.path.splitext(self.path)[0] if self.path else os.path.join(self.default_dir(), "page")
path, _ = QFileDialog.getSaveFileName(self, "Save text", base + ".txt", "Text (*.txt)")
if path:
with open(path, "w", encoding="utf-8") as f:
f.write(self.text + "\n")
self.set_message(f"saved {path}")
def copy_text(self):
QGuiApplication.clipboard().setText(self.view.text_range())
self.set_message("text copied to the clipboard")
def closeEvent(self, ev):
if self.scan_thread and self.scan_thread.isRunning():
+70
View File
@@ -0,0 +1,70 @@
"""Clean up a page for reading: flatten the paper to white, darken the text, and drop
bleed-through (the other side's print showing through, fainter than this side's ink).
The paper's own brightness is estimated per channel by closing the ink away (a max
filter wider than a stroke) and smoothing; dividing by it makes the paper 1.0 all over,
whatever the lighting or tint. On that scale ink and bleed-through separate by depth:
Otsu's threshold over the page splits the ink from everything lighter, and the white
point is set part way from it to the paper, so marks lighter than that (the bleed-through,
show-through texture, faint shadows) go to white and the ink is stretched to black. Within
a few pixels of the ink the white point stays at the paper: faint pixels there are the
ink's own soft edges and thin strokes, which would otherwise be eaten away.
Depth is the darkest channel, so coloured ink is ink too, and colour is kept by scaling
what each channel absorbs by one factor: blue ink comes out a deeper blue, grey ink grey.
"""
import cv2
import numpy as np
def paper(img, blank=None, stroke=40, k=4):
"""float32 estimate of the paper's colour under each pixel of a BGR or gray uint8 page.
`stroke`: a width in pixels larger than any ink stroke; `blank`: a bool mask of pixels
that are not page (left out). Done on a k-times smaller copy: the paper varies slowly."""
f = img.astype(np.float32)
if blank is not None and blank.any():
f[blank] = np.median(f[~blank], axis=0) # don't let the unscanned white lift the edges
small = cv2.resize(f, None, fx=1 / k, fy=1 / k, interpolation=cv2.INTER_AREA)
r = max(1, stroke // k)
small = cv2.dilate(small, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2 * r + 1, 2 * r + 1)))
small = cv2.GaussianBlur(small, (0, 0), r)
bg = cv2.resize(small, (img.shape[1], img.shape[0]), interpolation=cv2.INTER_LINEAR)
# Large dark areas (the table beside the page) are not paper: keep them dark.
floor = 0.6 * np.percentile(bg if blank is None else bg[~blank], 90, axis=0)
return np.maximum(bg, floor.astype(np.float32))
def enhance(img, blank=None, strength=0.25, edge=3):
"""The page with its paper flattened to white, ink stretched to black and marks fainter
than the ink (bleed-through) whitened. `strength` (0-1): how far the white point moves
from the paper towards the ink threshold; `edge`: how many pixels around the ink keep
their shading (its soft edges). Returns uint8 of the same shape; `blank` pixels are
left as they were."""
norm = img.astype(np.float32) / paper(img, blank)
# Lightness as the darkest channel, so coloured ink (blue absorbs red) counts as ink.
lum = norm.min(axis=2) if norm.ndim == 3 else norm
page = lum if blank is None else lum[~blank]
hist = np.clip(page * 255, 0, 255).astype(np.uint8)
t, _ = cv2.threshold(hist.reshape(-1, 1), 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
t /= 255
white = 1 - strength * (1 - t)
ink = page[page < t]
black = np.percentile(ink, 5) if ink.size else 0.0
black = min(black, white - 0.05)
# Next to ink, faint pixels are its anti-aliased edges and fine strokes, not bleed-through:
# there the white point stays at the paper, so strokes keep their full width and shading.
core = (lum < t).astype(np.uint8)
near = cv2.dilate(core, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2 * edge + 1,) * 2))
near = cv2.GaussianBlur(near.astype(np.float32), (0, 0), edge / 2)
white = white + (1 - white) * near
level = np.clip((lum - black) / (white - black), 0, 1)
if norm.ndim == 3:
# Scale what each channel absorbs by the same factor, the darkness's stretch: the
# ink keeps its hue (its channels' proportions) and gets deeper, never clipped.
absorbed = np.clip(1 - norm, 0, None)
k = (1 - level) / np.maximum(1 - lum, 1e-3)
level = 1 - absorbed * k[..., None]
out = (255 * np.clip(level, 0, 1)).round().astype(np.uint8)
if blank is not None:
out[blank] = img[blank]
return out
+82 -5
View File
@@ -4,8 +4,10 @@ LivePreview keeps a downscaled copy of the page: each frame is pasted at its cur
best pose as it is registered (the last PageGraph solution, extended by odometry), and
when a newer solution arrives the whole preview is redrawn from it, so drift visibly
snaps into place. LiveSession runs the periodic re-solve and publishes the latest frame,
the preview and a status dict to a viewer (see webview.py).
the preview and a status dict to a viewer (see webview.py), including live sensor readings
under "sensors" (see LiveSession.sensors).
"""
import bisect
import threading
import time
import traceback
@@ -13,9 +15,14 @@ import traceback
import numpy as np
import cv2
from .session import bright_level
from .stitching.canvas import Canvas, bbox, normalize, page_orient
from .stitching.filters import feather
from .stitching.geometry import corners, pose_matrix
from .stitching.pagegraph import SIGMA_ECC
MM_PER_PX = 0.0613 # frame pixel on the paper (the EEPROM's pixelSize); about 414 dpi
E6_PER_MS = 49.23 # exposure register units per millisecond
class LivePreview:
@@ -30,6 +37,7 @@ class LivePreview:
self.cur = None # pose of the last pasted frame
self.seen = None # solution the canvas was drawn from
self.version = 0
self.covered = 0 # preview pixels covered by the last render
def add(self, corrected):
"""Called after PageGraph.add with the corrected frame it returned. Cheap: pastes
@@ -78,6 +86,7 @@ class LivePreview:
return None
acc, ws, off, cur = self.c.acc.copy(), self.c.ws.copy(), self.c.off.copy(), self.cur
out, covered = normalize(acc, ws, background=background)
self.covered = int(np.count_nonzero(covered))
if out.ndim == 2:
out = cv2.cvtColor(out, cv2.COLOR_GRAY2BGR)
if cur is not None:
@@ -93,10 +102,15 @@ class LiveSession:
"""Per-scan live view: re-solves every `every` frames on a background thread and
publishes frame, preview and status to `viewer` a few times a second."""
def __init__(self, graph, viewer, every=45, fps=8):
def __init__(self, graph, viewer, every=45, fps=8, mm_per_px=MM_PER_PX):
self.g, self.viewer, self.every, self.period = graph, viewer, every, 1 / fps
self.mm = mm_per_px
self.preview = LivePreview(graph)
self.raw = None # latest captured frame (set by the capture loop)
self.raw = self.rec = None # latest captured frame and its sensor record (on_capture)
self.e6, self.times = 0, []
self.frame_of = [] # graph frame showing each processed capture (still ones are dropped)
self.dist, self.dist_done = 0.0, 0
viewer.set_status(sensors=None)
self._solving = True
self._closed = False
self._solver = threading.Thread(target=self._solve_loop, daemon=True)
@@ -104,8 +118,66 @@ class LiveSession:
self._solver.start()
self._render.start()
def on_capture(self, t, raw, rec, e6):
"""Called by the capture loop with each frame, its sensor record and the exposure
(E6) it was taken with."""
self.times.append(t)
self.raw, self.rec, self.e6 = raw, rec, e6
def on_frame(self, corrected):
self.preview.add(corrected)
"""Called with what PageGraph.add returned for each captured frame, in order."""
self.frame_of.append(self.g.n - 1)
if corrected is not None:
self.preview.add(corrected)
def sensors(self, window=0.3):
"""Live readings for the viewer, or None before the first registered frame:
heading: the mouse's rotation on the page since the scan started, in degrees,
clockwise as seen in the page preview;
velocity: (x, y) mm/s on the page (y down), over the last `window` seconds;
speed (mm/s), distance travelled (mm), offset (x, y) from the start (mm),
area: page covered so far (cm²);
tracking: share of the last 30 frame steps registered by image (the rest
only had the lasers, which happens when moving too fast or lifted);
lasers: the two laser sensors' surface quality (record bytes 14, 15);
exposure (ms), level: paper brightness (0-255), fps."""
g, times, rec, frame_of = self.g, self.times, self.rec, self.frame_of
k = len(frame_of) - 1 # newest processed capture
if k < 1 or rec is None:
return None
i = frame_of[k] # and the graph frame that shows it
c = np.array([g.w / 2, g.h / 2, 1.0])
def centre(k):
# Frame centre in page orientation (page_orient: transpose, then turn 180°),
# up to an offset: canvas (x, y) -> page (-y, -x).
x, y = (g.pose(k) @ c)[:2]
return np.array([-y, -x])
M = g.pose(i)
# The frame's x axis, (cos θ, sin θ) on the canvas, is (-sin θ, -cos θ) on the page:
# it points up at θ = 0 and turns by -θ.
heading = -np.degrees(np.arctan2(M[1, 0], M[0, 0]))
j = bisect.bisect_left(times, times[k] - window, 0, k)
p = centre(i)
dt = times[k] - times[j]
vel = (p - centre(frame_of[j])) * self.mm / dt if j < k and dt > 0 else np.zeros(2)
for _, _, T, _ in g.odo[self.dist_done:i]:
self.dist += float(np.hypot(*(T @ c - c)[:2])) * self.mm
self.dist_done = i
recent = g.odo[max(0, i - 30):i]
t = times[-1]
fps = (len(times) - 1 - bisect.bisect_left(times, t - 1.0)) / min(1.0, t) if t > 0 else 0.0
raw = self.raw
return {
"heading": float((heading + 180) % 360 - 180),
"velocity": [float(v) for v in vel], "speed": float(np.hypot(*vel)),
"distance": self.dist, "offset": [float(v) for v in (p - centre(0)) * self.mm],
"area": self.preview.covered * (self.mm / self.preview.s) ** 2 / 100,
"tracking": sum(e[3] == SIGMA_ECC for e in recent) / len(recent) if recent else 1.0,
"lasers": [int(rec.b14), int(rec.b15)],
"exposure": self.e6 / E6_PER_MS, "level": bright_level(raw) if raw is not None else 0.0,
"fps": float(fps),
}
def _solve_loop(self):
"""Re-solve every `every` frames, resting at least as long as the last solve took
@@ -142,7 +214,12 @@ class LiveSession:
img = self.preview.render()
if img is not None:
self.viewer.publish("page", img)
self.viewer.set_status(loops=len(self.g.loops))
try:
sensors = self.sensors()
except Exception: # a reading must never stop the preview
traceback.print_exc()
sensors = None
self.viewer.set_status(loops=len(self.g.loops), sensors=sensors)
time.sleep(max(0.0, self.period - (time.time() - t)))
def close(self):
+151 -15
View File
@@ -6,12 +6,16 @@ Latin script (English, Swedish, German, French, ...) as well as Chinese and Japa
Independent of the scanner hardware, like myscan.stitching.
lines = Ocr()(bgr) # [Line(box, text, score, edges)], box = 4x2 corner points
lines, flipped = Ocr().read(bgr) # ... and whether the page was upside down
text, spans = layout(lines) # plain text; spans[i] = (start, end) of lines[i] in it
joins = paragraph_joins(lines, text, spans)
reflow(text, joins) # the text with paragraphs on one line each
Each line knows where its characters are (from the recogniser's per-character boxes),
so text can be selected on the image: Line.locate maps a point to a character
position, Line.quad gives the outline of a range of characters.
"""
import re
import threading
from dataclasses import dataclass
@@ -30,12 +34,9 @@ class Line:
self.edges = np.linspace(0.0, 1.0, len(self.text) + 1)
@property
def cy(self):
return float(self.box[:, 1].mean())
@property
def left(self):
return float(self.box[:, 0].min())
def width(self):
"""Length along the box's top edge."""
return float(np.hypot(*(self.box[1] - self.box[0])))
@property
def height(self):
@@ -105,11 +106,25 @@ def char_edges(box, text, chars):
return np.clip(np.maximum.accumulate(edges), 0.0, 1.0)
def quality(lines):
"""Mean confidence per character."""
n = sum(len(ln.text) for ln in lines)
return sum(ln.score * len(ln.text) for ln in lines) / n if n else 0.0
class Ocr:
"""Lazily loads the engine (about 0.3 s) on first use; calls are serialised."""
"""Lazily loads the engine (about 0.3 s) on first use; calls are serialised.
RapidOCR's angle classifier, which flips each line it thinks is upside down, is off:
it judges lines one by one and now and then turns a perfectly upright line over,
reading it as garbage. All lines on a page face the same way, so the page is turned
as a whole instead: upright text reads at 0.97-1.0 confidence and upside-down text
at 0.55-0.8, so a page reading poorly is read again rotated 180 degrees, and the
better reading kept. That also covers pages turned 90 degrees, whose lines are cut
out turned one way, which is either right or upside down."""
def __init__(self, **params):
self.params = {"Global.log_level": "warning", **params}
self.params = {"Global.log_level": "warning", "Global.use_cls": False, **params}
self._engine, self._lock = None, threading.Lock()
def engine(self):
@@ -119,6 +134,19 @@ class Ocr:
return self._engine
def __call__(self, bgr):
return self.read(bgr)[0]
def read(self, bgr):
"""(lines, flipped): flipped if the page read better rotated 180 degrees, in
which case the boxes are in the coordinates of the rotated image."""
lines = self.lines(bgr)
if quality(lines) < 0.9:
turned = self.lines(np.ascontiguousarray(bgr[::-1, ::-1]))
if quality(turned) >= max(0.85, quality(lines)):
return turned, True
return lines, False
def lines(self, bgr):
with self._lock:
r = self.engine()(bgr, return_word_box=True, return_single_char_box=True)
if r.boxes is None:
@@ -131,18 +159,49 @@ class Ocr:
return out
def text_direction(lines):
"""Unit vector along the text lines: the length-weighted median direction of the
lines' top edges (short lines, whose angle is unreliable, only if there is nothing
else). A scan held at an angle has every line slanted the same way."""
if not lines:
return np.array([1.0, 0.0])
tops = np.array([ln.box[1] - ln.box[0] for ln in lines])
length = np.hypot(tops[:, 0], tops[:, 1])
long = length > 3 * np.array([ln.height for ln in lines])
if long.any():
tops, length = tops[long], length[long]
angle = np.arctan2(tops[:, 1], tops[:, 0])
order = np.argsort(angle)
cum = np.cumsum(length[order])
a = angle[order][np.searchsorted(cum, cum[-1] / 2)]
return np.array([np.cos(a), np.sin(a)])
def page_coords(lines):
"""(along, across) position of each line in the page's own frame, unaffected by the
scan's slant: where the line starts along the text direction, and the offset of its
centre perpendicular to it (the same anywhere along a straight line)."""
d = text_direction(lines)
n = np.array([-d[1], d[0]])
return [(float((ln.box @ d).min()), float(ln.box.mean(0) @ n)) for ln in lines]
def rows(lines):
"""Group lines into visual rows (top to bottom), each sorted left to right. A line
joins a row when its centre is within half a line height of the row's."""
"""Group lines into visual rows (top to bottom), each sorted left to right, in the
page's frame (page_coords). A line joins a row when its centre is within half a line
height of the row's. Returns (rows, the rows' centre offsets)."""
pos = {id(ln): p for ln, p in zip(lines, page_coords(lines))}
across = lambda ln: pos[id(ln)][1]
out = []
for ln in sorted(lines, key=lambda ln: ln.cy):
for ln in sorted(lines, key=across):
for row in out[-3:]:
if abs(ln.cy - np.mean([r.cy for r in row])) < 0.5 * min(ln.height, row[0].height):
if abs(across(ln) - np.mean([across(r) for r in row])) < 0.5 * min(ln.height, row[0].height):
row.append(ln)
break
else:
out.append([ln])
return [sorted(row, key=lambda ln: ln.left) for row in out]
return ([sorted(row, key=lambda ln: pos[id(ln)][0]) for row in out],
[float(np.mean([across(ln) for ln in row])) for row in out])
def layout(lines):
@@ -151,8 +210,7 @@ def layout(lines):
row are separated by a tab. Returns (text, spans), spans[i] = (start, end) of
lines[i] in text."""
index = {id(ln): i for i, ln in enumerate(lines)}
rs = rows(lines)
cys = [float(np.mean([ln.cy for ln in row])) for row in rs]
rs, cys = rows(lines)
steps = np.diff(cys)
pitch = float(np.median(steps)) if len(steps) else 0.0
spans, parts, pos = [None] * len(lines), [], 0
@@ -169,3 +227,81 @@ def layout(lines):
parts.append(ln.text)
pos += len(ln.text)
return "".join(parts), spans
def _extent(ln, o, d):
"""(start, end) of a line's box along direction d (unit vector) from point o."""
t = (ln.box - o) @ d
return float(t.min()), float(t.max())
# a list item's marker: "(a)", "a)", "1.", "iv)", a bullet or dash, then a space
MARKER = re.compile(r"(\(?[0-9A-Za-z]{1,3}[.)]|[•·▪◦*\-–])\s+")
def _continues(rs, k):
"""Whether the first line of row k + 1 continues the paragraph of the last line of row
k: similar text height, the next line isn't indented (except to line up with the
text after a list marker, a hanging indent), and line k is full, i.e. the next
line's first word would not have fitted in the space left at its end (which works
for justified and ragged-right text alike)."""
a, b = rs[k][-1], rs[k + 1][0]
h = min(a.height, b.height)
if not a.text.strip() or not b.text.strip() or abs(a.height - b.height) > 0.3 * max(a.height, b.height):
return False
o = a.box[0]
d = (a.box[1] - a.box[0]) / a.width
la, ra = _extent(a, o, d)
lb, _ = _extent(b, o, d)
if lb > la + 0.8 * h:
m = MARKER.match(a.text)
if not m or abs(lb - float((a.quad(m.end(), m.end())[0] - o) @ d)) > 0.8 * h:
return False
left = min(la, lb)
right = ra
for j in range(max(k - 2, 0), min(k + 4, len(rs))): # the paragraph's right margin
c = rs[j][-1]
if abs(c.height - h) < 0.3 * h:
lj, rj = _extent(c, o, d)
if abs(lj - left) < 3 * h:
right = max(right, rj)
_, e = b.word(0)
word = (b.edges[e] - b.edges[0]) * b.width
return right - ra < word + 0.5 * h
def paragraph_joins(lines, text, spans):
"""Where layout()'s text breaks a paragraph across lines, as edits for reflow():
(start, end, replacement) replacing a line break with a space, or removing it along
with the hyphen of a word split over two lines (the hyphen stays, without the
break, when the next line starts with a capital, as in "Nord-Amerika"). Only a row
holding a single line continues a paragraph, and only a single line or a wide last
item (as after a separately read section number) runs on into it; rows of several
short items are a table."""
index = {id(ln): i for i, ln in enumerate(lines)}
rs, _ = rows(lines)
out = []
for k in range(len(rs) - 1):
a, b = rs[k][-1], rs[k + 1][0]
if len(rs[k + 1]) != 1 or (len(rs[k]) != 1 and a.width < 10 * a.height):
continue
e = spans[index[id(a)]][1]
if text[e:spans[index[id(b)]][0]] != "\n" or not _continues(rs, k):
continue
if len(a.text) > 1 and a.text[-1] in "-\u00ad¬" and a.text[-2].isalpha():
out.append((e - 1, e + 1, "") if b.text[0].islower() else (e, e + 1, ""))
else:
out.append((e, e + 1, " "))
return out
def reflow(text, joins, a=0, b=None):
"""text[a:b] with the paragraph_joins() edits that lie inside it applied."""
b = len(text) if b is None else b
parts, pos = [], a
for s, e, r in joins:
if a <= s and e <= b:
parts += [text[pos:s], r]
pos = e
parts.append(text[pos:b])
return "".join(parts)
+9 -6
View File
@@ -17,7 +17,7 @@ from types import SimpleNamespace
import cv2
from . import calib
from .live import LiveSession
from .live import MM_PER_PX, LiveSession
from .session import ScanSession
from .stitching import Corrector, PageGraph
from .xu import XU
@@ -60,7 +60,7 @@ def wait_press(x, viewer=None, quit=None, poll=0.05):
return True
def scan_page(x, a, frames_dir=None, corrector=None, gains=None, viewer=None):
def scan_page(x, a, frames_dir=None, corrector=None, gains=None, viewer=None, mm_per_px=MM_PER_PX):
"""Capture until the button is pressed again. Frames are registered (and saved) on a
worker thread so capture never waits; returns (PageGraph or None, LiveSession or None)."""
g, live, rows, work, saves = None, None, [], queue.Queue(), queue.Queue()
@@ -87,14 +87,14 @@ def scan_page(x, a, frames_dir=None, corrector=None, gains=None, viewer=None):
if g is None:
g = PageGraph(im.shape, corrector)
if viewer:
live = LiveSession(g, viewer)
live = LiveSession(g, viewer, mm_per_px=mm_per_px)
th = threading.Thread(target=worker, daemon=True)
th.start()
work.put((n, im, rec.v))
if sv:
saves.put((n, im))
if live:
live.raw = im
live.on_capture(t, im, rec, s.e6)
if frames_dir:
rows.append([round(t, 4), n, rec.counter, *rec.v, rec.b14, rec.b15, rec.b16, rec.flags, s.e6,
round(float(im.mean()), 1), rec.raw.hex()])
@@ -112,7 +112,8 @@ def scan_page(x, a, frames_dir=None, corrector=None, gains=None, viewer=None):
if viewer:
viewer.set_status(t=t, frames=n + 1, backlog=work.qsize())
if t - last >= 1:
print(f"\r {t:5.1f}s {n + 1} frames ({g.n} registered) E6={s.e6}", end="", flush=True)
print(f"\r {t:5.1f}s {n + 1} frames ({g.n} stitched, {g.stats['still']} still) E6={s.e6}",
end="", flush=True)
last = t
print()
if viewer:
@@ -147,6 +148,7 @@ class Scanner:
self.corrector = Corrector(None if a.no_flat else calib.FlatField(cal),
None if a.no_undistort else calib.Undistort(cal))
self.gains = cal["gains_rgb"]
self.mm_per_px = cal["pixel_size_m"] * 1000
def wait_press(self, quit=None):
if self.viewer:
@@ -163,7 +165,8 @@ class Scanner:
frames_dir = os.path.join(a.outdir, name) if a.save_frames else None
if frames_dir:
os.makedirs(frames_dir)
g, live = scan_page(self.x, a, frames_dir, self.corrector, self.gains, viewer)
g, live = scan_page(self.x, a, frames_dir, self.corrector, self.gains, viewer,
self.mm_per_px)
if live:
live.stop_solving()
live.close()
+28 -3
View File
@@ -17,6 +17,9 @@ wrong copy. Here every frame keeps its own pose and the poses are solved togethe
pose; the rest (off the page, lifted, corrupt) only had the odometry and are left out,
as are frames that disagree with the mosaic of all the others.
Frames that show nothing new (the mouse lying still) are dropped on arrival: they cost
registration, solving and rendering time and add nothing to the page.
A pose M (3x3) maps frame pixels to canvas pixels; an edge (i, j, T) says M_j ~ M_i @ T.
"""
import os
@@ -37,6 +40,10 @@ SIGMA_ECC = 1.0 # px, edge registered by ECC
SIGMA_ODO = 6.0 # px, odometry edge from the motion prior only
SIGMA_LOOP = 1.5 # px, loop closure
PRUNE = 5.0 # drop loop edges whose residual exceeds this many sigmas
STILL_PX = 1.0 # odometry motion (px at the frame corners) below which a frame may be still
STILL_DIFF = 0.01 # and thumbnail difference below which it is (sensor noise is ~0.002,
# slow motion 0.01-0.03; relative to the mean, so exposure changes don't count)
THUMB = (80, 37) # thumbnail size for that test
class PageGraph:
@@ -56,10 +63,12 @@ class PageGraph:
self._hpf = {}
self.odo, self.bridges, self.loops = [], [], [] # edges: (i, j, T, sigma)
self.prev_v = None
self._thumb = None # of the last frame added
self._feat, self._tried = {}, set() # kept across solves
self._pool = None
self.solution = None # (poses (n, 3), indices of frames to draw)
self.stats = {"frames": 0, "odo_ecc": 0, "odo_laser": 0, "bridges": 0, "loop_tries": 0, "pruned": 0}
self.stats = {"frames": 0, "odo_ecc": 0, "odo_laser": 0, "bridges": 0, "loop_tries": 0, "pruned": 0,
"still": 0}
@property
def n(self):
@@ -76,13 +85,29 @@ class PageGraph:
self._feat[i] = register.detect(self._hp(i))
return self._feat[i]
def _still(self, thumb, v):
"""Whether a frame with this thumbnail and sensor values shows the same as the last
frame added: neither the odometry (without image shift) nor the image moved."""
T, _ = self.odometry.predict(self.prev_v, v, self.center)
q = np.array([[0, self.w, 0, self.w], [0, 0, self.h, self.h], [1, 1, 1, 1]])
if np.abs(T @ q - q).max() >= STILL_PX:
return False
return cv2.norm(thumb, self._thumb, cv2.NORM_L1) / thumb.size < STILL_DIFF
def add(self, frame, v):
"""frame: HxW or HxWx3 uint8 raw frame; v: its sensor values for the odometry (the
myScan record's six raw values). Returns the corrected float frame.
myScan record's six raw values). Returns the corrected float frame, or None if the
frame was dropped as still (see _still).
Safe to call while solve() runs on another thread."""
v = np.asarray(v)
thumb = cv2.resize(frame, THUMB, interpolation=cv2.INTER_AREA).astype(np.float32)
thumb = gray(thumb) / max(float(thumb.mean()), 1.0)
if self.n and self._still(thumb, v):
self.stats["still"] += 1
return None
self._thumb = thumb
f = self.correct(frame)
hs = cv2.resize(highpass_fast(gray(f)), self.half, interpolation=cv2.INTER_AREA)
v = np.asarray(v)
i = self.n
self.frames.append(frame)
self.hs.append(hs)