Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
04c10e58c7 | ||
|
|
eeea9c3b30 | ||
|
|
5f9ecca302 | ||
|
|
e9ab616161 | ||
|
|
0dd807a4e7 | ||
|
|
61b6cf70a7 | ||
|
|
16df509d01 | ||
|
|
2ef567580a | ||
|
|
a1942faa41 | ||
|
|
6631073be5 |
@@ -13,18 +13,28 @@ Linux driver tools for the Hama myScan (00052343) scanner mouse, a Dacuda design
|
|||||||
prior; another device can pass its own object with `predict()`), `register` (ECC, phase correlation, SIFT
|
prior; another device can pass its own object with `predict()`), `register` (ECC, phase correlation, SIFT
|
||||||
matching, overlap check), `canvas` (weighted canvases, page composition, parallel rendering), `posegraph`
|
matching, overlap check), `canvas` (weighted canvases, page composition, parallel rendering), `posegraph`
|
||||||
(robust pose-graph optimisation), `loops` (loop-closure candidates), and the two stitchers built from them:
|
(robust pose-graph optimisation), `loops` (loop-closure candidates), and the two stitchers built from them:
|
||||||
`pagegraph` (`PageGraph`: assembles the page from all frames at once with loop closures between passes;
|
`pagegraph` (`PageGraph`: assembles the page from all frames at once with loop closures between passes,
|
||||||
|
dropping frames taken while the mouse lies still;
|
||||||
used by `scan.py` and `stitch.py`) and `incremental` (`Stitcher`: frame by frame, `stitch.py --live`)
|
used by `scan.py` and `stitch.py`) and `incremental` (`Stitcher`: frame by frame, `stitch.py --live`)
|
||||||
- `tools/scan.py` — scan with the button: `python tools/scan.py` (press to start, press again to stop).
|
- `tools/scan.py` — scan with the button: `python tools/scan.py` (press to start, press again to stop).
|
||||||
`--view [PORT]` serves a live view at `http://<host>:8000/`: the camera, the page as it is stitched
|
`--view [PORT]` serves a live view at `http://<host>:8000/`: the camera, the page as it is stitched
|
||||||
(re-solved every 1.5 s, current position outlined), Start/Stop buttons, PNG/JPEG download of the finished
|
(re-solved every 1.5 s, current position outlined), Start/Stop buttons, PNG/JPEG download of the finished
|
||||||
page and a Screenshot button (camera + page + status as one PNG)
|
page and a Screenshot button (camera + page + status as one PNG)
|
||||||
- `tools/app.py` — desktop app (Qt/PySide6; Linux, Windows, macOS): live camera and page view while scanning
|
- `tools/app.py` — desktop app (Qt/PySide6; Linux, Windows, macOS): live camera and page view while scanning
|
||||||
(Start/Stop or the scan button, F5), then OCR of the finished page with RapidOCR (`myscan/ocr.py`;
|
(Start/Stop or the scan button, F5). Under the camera a dial shows how the mouse is turned on the page since the
|
||||||
multilingual PP-OCRv6 models bundled in the wheel, works offline). Recognised lines are outlined on the page by
|
scan started (a big "A", coloured by tracking), its direction and speed (arrow), and the live readings: speed,
|
||||||
|
distance, offset from the start, area covered, tracking (share of recent frames matched by image rather than
|
||||||
|
the lasers alone), the lasers' surface quality, exposure, paper brightness and frame rate. Then OCR of the finished page with RapidOCR (`myscan/ocr.py`;
|
||||||
|
multilingual PP-OCRv6 models bundled in the wheel, works offline; an upside-down page is detected and turned). Level (Ctrl+L, on by default) shows the page rotated so its lines
|
||||||
|
are horizontal, using the text direction found by OCR; switch it off to see the page as stitched. Save image saves
|
||||||
|
what is shown. Recognised lines are outlined on the page by
|
||||||
confidence, and the text is selected right on the page like in a PDF viewer: drag (in reading order across
|
confidence, and the text is selected right on the page like in a PDF viewer: drag (in reading order across
|
||||||
lines), double-click a word, triple-click a line, shift-click to extend, Ctrl+A; Ctrl+C or right-click copies.
|
lines), double-click a word, triple-click a line, shift-click to extend, Ctrl+A; Ctrl+C or right-click copies.
|
||||||
Middle-drag or Space+drag pans, the wheel zooms. Copy all and Save text (.txt) take the whole page. `python tools/app.py [image]` also opens and reads existing images (Open, or
|
With Paragraphs on (Ctrl+J, the default), copied text has each paragraph on one line:
|
||||||
|
line breaks inside a paragraph become spaces and words hyphenated across lines are rejoined; a line ends a
|
||||||
|
paragraph if the next line is indented or its first word would have fitted on it (works for justified and
|
||||||
|
ragged text), and rows with several columns are left alone. Right-click offers the other form. Middle-drag or
|
||||||
|
Space+drag pans, the wheel zooms. Copy all takes the whole page. `python tools/app.py [image]` also opens and reads existing images (Open, or
|
||||||
drag and drop), which is all it does on machines without the mouse
|
drag and drop), which is all it does on machines without the mouse
|
||||||
- `myscan/scanner.py` — the scan loop shared by `scan.py` and the app (finds the mouse's `/dev/videoN` itself)
|
- `myscan/scanner.py` — the scan loop shared by `scan.py` and the app (finds the mouse's `/dev/videoN` itself)
|
||||||
- `myscan/live.py`, `myscan/webview.py` — the live preview and its browser viewer (standard library only)
|
- `myscan/live.py`, `myscan/webview.py` — the live preview and its browser viewer (standard library only)
|
||||||
|
|||||||
+330
-66
@@ -4,7 +4,8 @@ Left, the camera and the page as it is being stitched (the same preview as the b
|
|||||||
viewer); after a scan the finished page is loaded and read with RapidOCR (myscan/ocr.py).
|
viewer); after a scan the finished page is loaded and read with RapidOCR (myscan/ocr.py).
|
||||||
Recognised lines are outlined on the page, coloured by confidence, and their text can be
|
Recognised lines are outlined on the page, coloured by confidence, and their text can be
|
||||||
selected right on the page (drag, double-click a word, triple-click a line) and copied;
|
selected right on the page (drag, double-click a word, triple-click a line) and copied;
|
||||||
all of it can be copied or saved. Images can also be opened or dropped onto the window,
|
all of it can be copied or saved. Enhance whitens the paper, darkens the text and drops
|
||||||
|
bleed-through from the other side (myscan/enhance.py); OCR and Save use what is shown. Images can also be opened or dropped onto the window,
|
||||||
so OCR works on any platform; scanning needs Linux and the mouse.
|
so OCR works on any platform; scanning needs Linux and the mouse.
|
||||||
|
|
||||||
Run: python tools/app.py [image]
|
Run: python tools/app.py [image]
|
||||||
@@ -19,23 +20,43 @@ import traceback
|
|||||||
import cv2
|
import cv2
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from PySide6.QtCore import QObject, QPointF, QRectF, Qt, QThread, QTimer, Signal
|
from PySide6.QtCore import QObject, QPointF, QRectF, Qt, QThread, QTimer, Signal
|
||||||
from PySide6.QtGui import (QAction, QBrush, QClipboard, QColor, QGuiApplication, QImage, QKeySequence, QPainter,
|
from PySide6.QtGui import (QAction, QActionGroup, QBrush, QClipboard, QColor, QFont, QGuiApplication, QImage, QKeySequence,
|
||||||
QPainterPath, QPen, QPixmap, QPolygonF)
|
QPainter, QPainterPath, QPen, QPixmap, QPolygonF)
|
||||||
from PySide6.QtWidgets import (QApplication, QFileDialog, QGraphicsPathItem, QGraphicsPixmapItem,
|
from PySide6.QtWidgets import (QApplication, QFileDialog, QGraphicsPathItem, QGraphicsPixmapItem,
|
||||||
QGraphicsPolygonItem, QGraphicsScene, QGraphicsView, QLabel, QMainWindow, QMenu,
|
QGraphicsPolygonItem, QGraphicsRectItem, QGraphicsScene, QGraphicsView, QLabel, QMainWindow, QMenu,
|
||||||
QMessageBox, QSizePolicy, QSplitter, QToolBar, QVBoxLayout, QWidget)
|
QMessageBox, QSizePolicy, QSplitter, QToolBar, QToolButton, QVBoxLayout, QWidget)
|
||||||
|
|
||||||
from .ocr import Ocr, layout
|
from .enhance import enhance
|
||||||
|
from .ocr import Ocr, layout, paragraph_joins, reflow, text_direction
|
||||||
|
|
||||||
IMAGE_FILTER = "Images (*.png *.jpg *.jpeg *.tif *.tiff *.bmp *.webp)"
|
IMAGE_FILTER = "Images (*.png *.jpg *.jpeg *.tif *.tiff *.bmp *.webp)"
|
||||||
|
# Enhance strengths (see enhance.enhance): how much fainter than the ink a mark must be to
|
||||||
|
# be whitened as bleed-through. Stronger removes darker bleed-through and more faint lines.
|
||||||
|
ENHANCE_LEVELS = (("Light", 0.15), ("Normal", 0.25), ("Strong", 0.5))
|
||||||
|
|
||||||
|
|
||||||
def to_qimage(img):
|
def to_qimage(img, transparent=None):
|
||||||
"""BGR or grayscale uint8 array -> QImage (a copy)."""
|
"""BGR or grayscale uint8 array -> QImage (a copy); pixels where the bool mask
|
||||||
|
`transparent` is set get alpha 0."""
|
||||||
img = np.ascontiguousarray(img)
|
img = np.ascontiguousarray(img)
|
||||||
h, w = img.shape[:2]
|
h, w = img.shape[:2]
|
||||||
|
if transparent is None:
|
||||||
fmt = QImage.Format_Grayscale8 if img.ndim == 2 else QImage.Format_BGR888
|
fmt = QImage.Format_Grayscale8 if img.ndim == 2 else QImage.Format_BGR888
|
||||||
return QImage(img.data, w, h, img.strides[0], fmt).copy()
|
return QImage(img.data, w, h, img.strides[0], fmt).copy()
|
||||||
|
bgra = cv2.cvtColor(img, cv2.COLOR_GRAY2BGRA if img.ndim == 2 else cv2.COLOR_BGR2BGRA)
|
||||||
|
bgra[transparent, 3] = 0
|
||||||
|
return QImage(bgra.data, w, h, bgra.strides[0], QImage.Format_ARGB32).copy()
|
||||||
|
|
||||||
|
|
||||||
|
def unscanned(img):
|
||||||
|
"""Mask of a stitched page's unscanned area: the pure-white regions touching the
|
||||||
|
border (the stitcher fills uncovered pixels with 255 and crops to the scanned area).
|
||||||
|
Paper that clips to 255 inside the page is not connected to the border and stays."""
|
||||||
|
white = ((img == 255).all(axis=-1) if img.ndim == 3 else img == 255).astype(np.uint8)
|
||||||
|
_, labels = cv2.connectedComponents(white, connectivity=4)
|
||||||
|
edge = np.concatenate([labels[0], labels[-1], labels[:, 0], labels[:, -1]])
|
||||||
|
edge = np.unique(edge[edge > 0])
|
||||||
|
return np.isin(labels, edge) if len(edge) else None
|
||||||
|
|
||||||
|
|
||||||
def imread(path):
|
def imread(path):
|
||||||
@@ -50,6 +71,17 @@ def imwrite(path, img):
|
|||||||
buf.tofile(path)
|
buf.tofile(path)
|
||||||
|
|
||||||
|
|
||||||
|
def level_image(img, angle):
|
||||||
|
"""The image rotated by -angle degrees (clockwise in image coordinates, y down), as
|
||||||
|
PageView shows it levelled, on a white canvas big enough to hold all of it."""
|
||||||
|
h, w = img.shape[:2]
|
||||||
|
m = cv2.getRotationMatrix2D((w / 2, h / 2), angle, 1.0)
|
||||||
|
c, s = abs(m[0, 0]), abs(m[0, 1])
|
||||||
|
W, H = int(np.ceil(w * c + h * s)), int(np.ceil(w * s + h * c))
|
||||||
|
m[:, 2] += ((W - w) / 2, (H - h) / 2)
|
||||||
|
return cv2.warpAffine(img, m, (W, H), flags=cv2.INTER_CUBIC, borderValue=(255, 255, 255))
|
||||||
|
|
||||||
|
|
||||||
def log_error(msg):
|
def log_error(msg):
|
||||||
"""Print an error, with the traceback of the exception being handled if any."""
|
"""Print an error, with the traceback of the exception being handled if any."""
|
||||||
print(f"myScan: {msg}", file=sys.stderr, flush=True)
|
print(f"myScan: {msg}", file=sys.stderr, flush=True)
|
||||||
@@ -136,7 +168,7 @@ class ScanThread(QThread):
|
|||||||
|
|
||||||
|
|
||||||
class OcrThread(QThread):
|
class OcrThread(QThread):
|
||||||
done = Signal(object, float) # lines, seconds
|
done = Signal(object, bool, float) # lines, page upside down, seconds
|
||||||
failed = Signal(str)
|
failed = Signal(str)
|
||||||
|
|
||||||
def __init__(self, ocr, img):
|
def __init__(self, ocr, img):
|
||||||
@@ -146,7 +178,7 @@ class OcrThread(QThread):
|
|||||||
def run(self):
|
def run(self):
|
||||||
t = time.time()
|
t = time.time()
|
||||||
try:
|
try:
|
||||||
self.done.emit(self.ocr(self.img), time.time() - t)
|
self.done.emit(*self.ocr.read(self.img), time.time() - t)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
log_error(f"OCR failed: {e}")
|
log_error(f"OCR failed: {e}")
|
||||||
self.failed.emit(str(e))
|
self.failed.emit(str(e))
|
||||||
@@ -169,8 +201,14 @@ class LineItem(QGraphicsPolygonItem):
|
|||||||
class PageView(QGraphicsView):
|
class PageView(QGraphicsView):
|
||||||
"""Zoomable page whose recognised text can be selected like in a PDF viewer: drag to
|
"""Zoomable page whose recognised text can be selected like in a PDF viewer: drag to
|
||||||
select in reading order, double-click for a word, triple-click for a line, shift-click
|
select in reading order, double-click for a word, triple-click for a line, shift-click
|
||||||
to extend. Middle-drag (or Space + drag, or any drag before there is text) pans. Keeps
|
to extend. Copied text either keeps the page's line breaks or, with `paragraphs` set,
|
||||||
the page fitted to the window until the user zooms."""
|
has each paragraph on one line (see ocr.paragraph_joins). Middle-drag (or Space +
|
||||||
|
drag, or any drag before there is text) pans. Keeps the page fitted to the window
|
||||||
|
until the user zooms.
|
||||||
|
|
||||||
|
The page, its line boxes and the selection are children of one item, which is
|
||||||
|
rotated to level the text (set_angle, set_level), so selection works the same either
|
||||||
|
way: everything is in image coordinates under that item."""
|
||||||
selectionChanged = Signal(str)
|
selectionChanged = Signal(str)
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
@@ -180,35 +218,57 @@ class PageView(QGraphicsView):
|
|||||||
self.setTransformationAnchor(QGraphicsView.AnchorUnderMouse)
|
self.setTransformationAnchor(QGraphicsView.AnchorUnderMouse)
|
||||||
self.setBackgroundBrush(QColor(30, 32, 36))
|
self.setBackgroundBrush(QColor(30, 32, 36))
|
||||||
self.setMouseTracking(True)
|
self.setMouseTracking(True)
|
||||||
self.pixmap = QGraphicsPixmapItem()
|
self.page = QGraphicsRectItem()
|
||||||
|
self.page.setPen(QPen(Qt.NoPen))
|
||||||
|
self.scene().addItem(self.page)
|
||||||
|
self.pixmap = QGraphicsPixmapItem(self.page)
|
||||||
self.pixmap.setTransformationMode(Qt.SmoothTransformation)
|
self.pixmap.setTransformationMode(Qt.SmoothTransformation)
|
||||||
self.scene().addItem(self.pixmap)
|
self.sel_item = QGraphicsPathItem(self.page)
|
||||||
self.sel_item = QGraphicsPathItem()
|
|
||||||
self.sel_item.setPen(QPen(Qt.NoPen))
|
self.sel_item.setPen(QPen(Qt.NoPen))
|
||||||
self.sel_item.setBrush(QColor(30, 110, 230, 90))
|
self.sel_item.setBrush(QColor(30, 110, 230, 90))
|
||||||
self.sel_item.setZValue(2)
|
self.sel_item.setZValue(2)
|
||||||
self.scene().addItem(self.sel_item)
|
self.angle, self.level = 0.0, True
|
||||||
self.items_, self.auto_fit, self.boxes_visible = [], True, True
|
self.items_, self.auto_fit, self.boxes_visible = [], True, True
|
||||||
self.lines, self.spans, self.text = [], [], ""
|
self.lines, self.spans, self.text, self.joins, self.paragraphs = [], [], "", [], True
|
||||||
self.sel, self.anchor, self.unit = (0, 0), None, None # unit: drag extends by "word"/"line"
|
self.sel, self.anchor, self.unit = (0, 0), None, None # unit: drag extends by "word"/"line"
|
||||||
self.pan_from, self.space, self.last_double = None, False, None
|
self.pan_from, self.space, self.last_double = None, False, None
|
||||||
|
|
||||||
def set_image(self, qimg, keep_view=False):
|
def set_image(self, qimg, keep_view=False):
|
||||||
self.pixmap.setPixmap(QPixmap.fromImage(qimg))
|
self.pixmap.setPixmap(QPixmap.fromImage(qimg))
|
||||||
self.scene().setSceneRect(QRectF(self.pixmap.pixmap().rect()))
|
|
||||||
if not keep_view:
|
if not keep_view:
|
||||||
self.auto_fit = True
|
self.auto_fit = True
|
||||||
|
self.place()
|
||||||
|
|
||||||
|
def set_angle(self, angle):
|
||||||
|
"""The text's slant in degrees (clockwise, as in image coordinates)."""
|
||||||
|
self.angle = angle
|
||||||
|
self.place()
|
||||||
|
|
||||||
|
def set_level(self, on):
|
||||||
|
self.level = on
|
||||||
|
self.place()
|
||||||
|
|
||||||
|
def place(self):
|
||||||
|
"""Rotate the page to level it (or not), and fit the scene around it."""
|
||||||
|
r = self.pixmap.boundingRect()
|
||||||
|
self.page.setTransformOriginPoint(r.center())
|
||||||
|
self.page.setRotation(-self.angle if self.level else 0.0)
|
||||||
|
self.scene().setSceneRect(self.page_rect())
|
||||||
if self.auto_fit:
|
if self.auto_fit:
|
||||||
self.fit()
|
self.fit()
|
||||||
|
|
||||||
|
def page_rect(self):
|
||||||
|
return self.page.mapRectToScene(self.pixmap.boundingRect())
|
||||||
|
|
||||||
def set_lines(self, lines, text="", spans=()):
|
def set_lines(self, lines, text="", spans=()):
|
||||||
for it in self.items_:
|
for it in self.items_:
|
||||||
self.scene().removeItem(it)
|
self.scene().removeItem(it)
|
||||||
self.items_ = [LineItem(ln.box, ln.score) for ln in lines]
|
self.items_ = [LineItem(ln.box, ln.score) for ln in lines]
|
||||||
for it in self.items_:
|
for it in self.items_:
|
||||||
it.setVisible(self.boxes_visible)
|
it.setVisible(self.boxes_visible)
|
||||||
self.scene().addItem(it)
|
it.setParentItem(self.page)
|
||||||
self.lines, self.text, self.spans = lines, text, list(spans)
|
self.lines, self.text, self.spans = lines, text, list(spans)
|
||||||
|
self.joins = paragraph_joins(lines, text, self.spans) if lines else []
|
||||||
self.sel, self.anchor = None, None
|
self.sel, self.anchor = None, None
|
||||||
self.select(0, 0)
|
self.select(0, 0)
|
||||||
|
|
||||||
@@ -221,7 +281,8 @@ class PageView(QGraphicsView):
|
|||||||
|
|
||||||
def hit(self, scene_pos):
|
def hit(self, scene_pos):
|
||||||
"""(line index, distance in pixels, text position) of the line nearest a scene point."""
|
"""(line index, distance in pixels, text position) of the line nearest a scene point."""
|
||||||
p = (scene_pos.x(), scene_pos.y())
|
q = self.page.mapFromScene(scene_pos)
|
||||||
|
p = (q.x(), q.y())
|
||||||
return min(((i, *ln.locate(p)) for i, ln in enumerate(self.lines)), key=lambda h: h[1])
|
return min(((i, *ln.locate(p)) for i, ln in enumerate(self.lines)), key=lambda h: h[1])
|
||||||
|
|
||||||
def position(self, scene_pos, unit=None):
|
def position(self, scene_pos, unit=None):
|
||||||
@@ -253,10 +314,20 @@ class PageView(QGraphicsView):
|
|||||||
def select_all(self):
|
def select_all(self):
|
||||||
self.select(0, len(self.text))
|
self.select(0, len(self.text))
|
||||||
|
|
||||||
def selected_text(self):
|
def text_range(self, a=0, b=None, paragraphs=None):
|
||||||
a, b = self.sel
|
"""The text from a to b (positions in self.text), reflowed into paragraphs if
|
||||||
|
`paragraphs` (default: self.paragraphs)."""
|
||||||
|
if self.paragraphs if paragraphs is None else paragraphs:
|
||||||
|
return reflow(self.text, self.joins, a, b)
|
||||||
return self.text[a:b]
|
return self.text[a:b]
|
||||||
|
|
||||||
|
def selected_text(self, paragraphs=None):
|
||||||
|
return self.text_range(*self.sel, paragraphs=paragraphs)
|
||||||
|
|
||||||
|
def set_paragraphs(self, on):
|
||||||
|
self.paragraphs = on
|
||||||
|
self.selectionChanged.emit(self.selected_text())
|
||||||
|
|
||||||
def drag_to(self, scene_pos):
|
def drag_to(self, scene_pos):
|
||||||
a, b = self.position(scene_pos, self.unit)
|
a, b = self.position(scene_pos, self.unit)
|
||||||
a0, b0 = self.anchor
|
a0, b0 = self.anchor
|
||||||
@@ -334,22 +405,25 @@ class PageView(QGraphicsView):
|
|||||||
if not self.lines:
|
if not self.lines:
|
||||||
return
|
return
|
||||||
m = QMenu(self)
|
m = QMenu(self)
|
||||||
m.addAction("Copy", self.copy).setEnabled(self.sel[0] != self.sel[1])
|
has_sel = self.sel[0] != self.sel[1]
|
||||||
m.addAction("Copy all text", lambda: QGuiApplication.clipboard().setText(self.text))
|
other = "Copy with the page's line breaks" if self.paragraphs else "Copy as paragraphs"
|
||||||
|
m.addAction("Copy", self.copy).setEnabled(has_sel)
|
||||||
|
m.addAction(other, lambda: self.copy(not self.paragraphs)).setEnabled(has_sel)
|
||||||
|
m.addAction("Copy all text", lambda: QGuiApplication.clipboard().setText(self.text_range()))
|
||||||
m.addSeparator()
|
m.addSeparator()
|
||||||
m.addAction("Select all", self.select_all)
|
m.addAction("Select all", self.select_all)
|
||||||
m.exec(ev.globalPos())
|
m.exec(ev.globalPos())
|
||||||
|
|
||||||
def copy(self):
|
def copy(self, paragraphs=None):
|
||||||
if self.selected_text():
|
if text := self.selected_text(paragraphs):
|
||||||
QGuiApplication.clipboard().setText(self.selected_text())
|
QGuiApplication.clipboard().setText(text)
|
||||||
|
|
||||||
# ---- zoom ----
|
# ---- zoom ----
|
||||||
|
|
||||||
def fit(self):
|
def fit(self):
|
||||||
self.auto_fit = True
|
self.auto_fit = True
|
||||||
if not self.pixmap.pixmap().isNull():
|
if not self.pixmap.pixmap().isNull():
|
||||||
self.fitInView(self.pixmap, Qt.KeepAspectRatio)
|
self.fitInView(self.page_rect(), Qt.KeepAspectRatio)
|
||||||
|
|
||||||
def zoom(self, f):
|
def zoom(self, f):
|
||||||
self.auto_fit = False
|
self.auto_fit = False
|
||||||
@@ -389,6 +463,102 @@ class ImageLabel(QLabel):
|
|||||||
Qt.SmoothTransformation))
|
Qt.SmoothTransformation))
|
||||||
|
|
||||||
|
|
||||||
|
class SensorDial(QWidget):
|
||||||
|
"""Live readings while scanning (LiveSession.sensors): a dial with a big "A" turned as
|
||||||
|
the mouse is turned on the page (upright: as at the start of the scan; the page preview
|
||||||
|
is drawn in that orientation), an arrow for the direction and speed of motion, and the
|
||||||
|
other readings listed below it."""
|
||||||
|
FULL_SPEED = 50.0 # mm/s for a full-length arrow; the lasers start to miscount near this
|
||||||
|
ROWS = 10
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__()
|
||||||
|
self.s = None
|
||||||
|
self.setMinimumSize(120, 160)
|
||||||
|
self.setSizePolicy(QSizePolicy.Preferred, QSizePolicy.Fixed)
|
||||||
|
self.setFixedHeight(200 + 17 * self.ROWS)
|
||||||
|
|
||||||
|
def set_sensors(self, s):
|
||||||
|
self.s = s
|
||||||
|
self.update()
|
||||||
|
|
||||||
|
def rows(self):
|
||||||
|
s = self.s
|
||||||
|
quality = lambda f: "#2ea043" if f >= 0.8 else "#db8c00" if f >= 0.5 else "#d62828"
|
||||||
|
fast = s["speed"] > 0.8 * self.FULL_SPEED
|
||||||
|
return [
|
||||||
|
("Heading", f"{s['heading']:+.1f}°", None),
|
||||||
|
("Speed", f"{s['speed']:.0f} mm/s", "#db8c00" if fast else None),
|
||||||
|
("Distance", f"{s['distance'] / 10:.1f} cm", None),
|
||||||
|
("Offset", "{:+.0f}, {:+.0f} mm".format(*s["offset"]), None),
|
||||||
|
("Area", f"{s['area']:.1f} cm²", None),
|
||||||
|
("Tracking", f"{s['tracking']:.0%}", quality(s["tracking"])),
|
||||||
|
("Lasers", "{} / {}".format(*s["lasers"]), None),
|
||||||
|
("Exposure", f"{s['exposure']:.2f} ms", None),
|
||||||
|
("Paper level", f"{s['level']:.0f} / 255", None),
|
||||||
|
("Frames", f"{s['fps']:.0f}/s", None),
|
||||||
|
]
|
||||||
|
|
||||||
|
def paintEvent(self, ev):
|
||||||
|
p = QPainter(self)
|
||||||
|
p.setRenderHint(QPainter.Antialiasing)
|
||||||
|
p.fillRect(self.rect(), QColor(30, 32, 36))
|
||||||
|
grey, light = QColor(139, 145, 154), QColor(220, 222, 226)
|
||||||
|
s = self.s
|
||||||
|
r = min(self.width() - 16, 184) / 2
|
||||||
|
cx, cy = self.width() / 2, 8 + r
|
||||||
|
p.save()
|
||||||
|
p.translate(cx, cy)
|
||||||
|
p.setPen(QPen(QColor(70, 74, 82), 1.5))
|
||||||
|
p.setBrush(QColor(24, 26, 30))
|
||||||
|
p.drawEllipse(QPointF(0, 0), r, r)
|
||||||
|
for deg in range(0, 360, 15): # ticks; long ones every 90°, page up at the top
|
||||||
|
p.save()
|
||||||
|
p.rotate(deg)
|
||||||
|
p.setPen(QPen(light if deg == 0 else grey, 2 if deg % 90 == 0 else 1))
|
||||||
|
p.drawLine(QPointF(0, -r), QPointF(0, -r + (9 if deg % 90 == 0 else 5)))
|
||||||
|
p.restore()
|
||||||
|
if s:
|
||||||
|
p.save()
|
||||||
|
p.rotate(s["heading"])
|
||||||
|
f = QFont(self.font())
|
||||||
|
f.setPixelSize(int(r * 1.2))
|
||||||
|
f.setBold(True)
|
||||||
|
glyph = QPainterPath()
|
||||||
|
glyph.addText(0, 0, f, "A")
|
||||||
|
glyph.translate(-glyph.boundingRect().center()) # turn about the letter's own centre
|
||||||
|
colour = QColor("#2ea043" if s["tracking"] >= 0.8 else "#db8c00" if s["tracking"] >= 0.5 else "#d62828")
|
||||||
|
p.setPen(Qt.NoPen)
|
||||||
|
p.setBrush(colour)
|
||||||
|
p.drawPath(glyph)
|
||||||
|
p.restore()
|
||||||
|
vx, vy = s["velocity"]
|
||||||
|
k = min(1.0, s["speed"] / self.FULL_SPEED) * (r - 6) / max(s["speed"], 1e-9)
|
||||||
|
if s["speed"] >= 1:
|
||||||
|
tip = QPointF(vx * k, vy * k)
|
||||||
|
p.setPen(QPen(QColor(90, 170, 255), 3, Qt.SolidLine, Qt.RoundCap))
|
||||||
|
p.drawLine(QPointF(0, 0), tip)
|
||||||
|
a = np.arctan2(vy, vx)
|
||||||
|
head = [tip + QPointF(10 * np.cos(a + d), 10 * np.sin(a + d)) for d in (np.pi * 0.85, -np.pi * 0.85)]
|
||||||
|
p.setBrush(QColor(90, 170, 255))
|
||||||
|
p.drawPolygon(QPolygonF([tip, *head]))
|
||||||
|
else:
|
||||||
|
p.setPen(grey)
|
||||||
|
p.drawText(QRectF(-r, -r, 2 * r, 2 * r), Qt.AlignCenter, "scan to see\nthe mouse's angle")
|
||||||
|
p.restore()
|
||||||
|
y = 16 + 2 * r
|
||||||
|
f = QFont(self.font())
|
||||||
|
f.setPixelSize(12)
|
||||||
|
p.setFont(f)
|
||||||
|
for name, value, colour in (self.rows() if s else ()):
|
||||||
|
p.setPen(grey)
|
||||||
|
p.drawText(QRectF(8, y, self.width() - 16, 17), Qt.AlignLeft | Qt.AlignVCenter, name)
|
||||||
|
p.setPen(QColor(colour) if colour else light)
|
||||||
|
p.drawText(QRectF(8, y, self.width() - 16, 17), Qt.AlignRight | Qt.AlignVCenter, value)
|
||||||
|
y += 17
|
||||||
|
p.end()
|
||||||
|
|
||||||
|
|
||||||
def pane(title, widget):
|
def pane(title, widget):
|
||||||
w = QWidget()
|
w = QWidget()
|
||||||
lay = QVBoxLayout(w)
|
lay = QVBoxLayout(w)
|
||||||
@@ -410,8 +580,12 @@ class MainWindow(QMainWindow):
|
|||||||
self.opts = opts
|
self.opts = opts
|
||||||
self.ocr = Ocr()
|
self.ocr = Ocr()
|
||||||
self.image, self.path, self.lines, self.text = None, None, [], ""
|
self.image, self.path, self.lines, self.text = None, None, [], ""
|
||||||
|
self.blank = None # unscanned-area mask of the shown page (set_page)
|
||||||
|
self.enhanced = None # self.image through enhance(), made when first shown
|
||||||
|
self.enhance_strength = dict(ENHANCE_LEVELS)["Normal"]
|
||||||
self.page_title = "Page"
|
self.page_title = "Page"
|
||||||
self.ocr_thread, self.scan_thread, self.scanning = None, None, False
|
self.ocr_thread, self.scan_thread, self.scanning = None, None, False
|
||||||
|
self.page_id = 0 # bumped for every new page, so a slow OCR of an old one is not applied
|
||||||
self.bridge = Bridge()
|
self.bridge = Bridge()
|
||||||
self.bridge.frame.connect(self.on_frame)
|
self.bridge.frame.connect(self.on_frame)
|
||||||
self.bridge.page.connect(self.on_preview)
|
self.bridge.page.connect(self.on_preview)
|
||||||
@@ -422,7 +596,6 @@ class MainWindow(QMainWindow):
|
|||||||
if self.device is None:
|
if self.device is None:
|
||||||
self.act_scan.setEnabled(False)
|
self.act_scan.setEnabled(False)
|
||||||
self.act_scan.setToolTip("No myScan found (scanning needs Linux and the mouse plugged in)")
|
self.act_scan.setToolTip("No myScan found (scanning needs Linux and the mouse plugged in)")
|
||||||
self.camera_pane.hide()
|
|
||||||
self.set_message("No scanner found: open or drop an image to read it")
|
self.set_message("No scanner found: open or drop an image to read it")
|
||||||
else:
|
else:
|
||||||
self.start_scanner()
|
self.start_scanner()
|
||||||
@@ -446,40 +619,72 @@ class MainWindow(QMainWindow):
|
|||||||
tb.addAction(a)
|
tb.addAction(a)
|
||||||
return a
|
return a
|
||||||
|
|
||||||
|
self.act_camera = act("Camera", lambda on: self.camera_pane.setVisible(on), "Ctrl+K",
|
||||||
|
"Show the camera and the live sensor readings", True)
|
||||||
|
tb.addSeparator()
|
||||||
self.act_scan = act("Start scan", self.toggle_scan, "F5", "Start or stop a scan (or press the mouse's scan button)")
|
self.act_scan = act("Start scan", self.toggle_scan, "F5", "Start or stop a scan (or press the mouse's scan button)")
|
||||||
tb.addSeparator()
|
tb.addSeparator()
|
||||||
act("Open…", self.open_image, QKeySequence.Open, "Open an image to read")
|
act("Open…", self.open_image, QKeySequence.Open, "Open an image to read")
|
||||||
self.act_save = act("Save image…", self.save_image, QKeySequence.Save, "Save the page image")
|
self.act_save = act("Save image…", self.save_image, QKeySequence.Save, "Save the page image")
|
||||||
tb.addSeparator()
|
tb.addSeparator()
|
||||||
self.act_ocr = act("Read text", self.run_ocr, "Ctrl+R", "Run OCR on the page")
|
|
||||||
self.act_auto = act("Auto", lambda: None, tip="Read the text after every scan", checkable=True)
|
|
||||||
self.act_auto.setChecked(True)
|
|
||||||
self.act_copy = act("Copy all", self.copy_text, "Ctrl+Shift+C", "Copy all text to the clipboard")
|
self.act_copy = act("Copy all", self.copy_text, "Ctrl+Shift+C", "Copy all text to the clipboard")
|
||||||
self.act_savetxt = act("Save text…", self.save_text, "Ctrl+Shift+S", "Save the text as a .txt file")
|
self.act_para = act("Paragraphs", self.view_paragraphs, "Ctrl+J",
|
||||||
|
"Copy each paragraph as one line, undoing the page's line breaks and "
|
||||||
|
"hyphenation (off: keep the lines as on the page)", True)
|
||||||
|
self.act_para.setChecked(True)
|
||||||
tb.addSeparator()
|
tb.addSeparator()
|
||||||
|
self.act_level = act("Level", self.set_level, "Ctrl+L",
|
||||||
|
"Show the page rotated so its lines are horizontal (off: as stitched); "
|
||||||
|
"Save image saves what is shown", True)
|
||||||
|
self.act_level.setChecked(True)
|
||||||
|
self.act_enhance = act("Enhance", self.set_enhance, "Ctrl+E",
|
||||||
|
"Whiten the paper, darken the text and remove bleed-through from the "
|
||||||
|
"other side of the page; the text is read again from what is shown. The arrow "
|
||||||
|
"sets how strongly: stronger removes darker bleed-through, but also faint lines",
|
||||||
|
True)
|
||||||
|
levels = QMenu(self)
|
||||||
|
group = QActionGroup(self)
|
||||||
|
for name, strength in ENHANCE_LEVELS:
|
||||||
|
a = levels.addAction(name, lambda strength=strength: self.set_enhance_strength(strength))
|
||||||
|
a.setCheckable(True)
|
||||||
|
a.setChecked(strength == self.enhance_strength)
|
||||||
|
group.addAction(a)
|
||||||
|
button = tb.widgetForAction(self.act_enhance)
|
||||||
|
button.setMenu(levels)
|
||||||
|
button.setPopupMode(QToolButton.MenuButtonPopup)
|
||||||
self.act_boxes = act("Boxes", lambda on: self.view.show_boxes(on), "Ctrl+B", "Show recognised lines on the page", True)
|
self.act_boxes = act("Boxes", lambda on: self.view.show_boxes(on), "Ctrl+B", "Show recognised lines on the page", True)
|
||||||
self.act_boxes.setChecked(True)
|
self.act_boxes.setChecked(True)
|
||||||
act("Fit", lambda: self.view.fit(), "Ctrl+0", "Fit the page to the window")
|
act("Fit", lambda: self.view.fit(), "Ctrl+0", "Fit the page to the window")
|
||||||
act("Zoom +", lambda: self.view.zoom(1.25), QKeySequence.ZoomIn)
|
for a in (self.act_save, self.act_copy):
|
||||||
act("Zoom −", lambda: self.view.zoom(0.8), QKeySequence.ZoomOut)
|
|
||||||
for a in (self.act_save, self.act_ocr, self.act_copy, self.act_savetxt):
|
|
||||||
a.setEnabled(False)
|
a.setEnabled(False)
|
||||||
|
|
||||||
self.camera = ImageLabel("camera")
|
self.camera = ImageLabel("camera")
|
||||||
|
self.dial = SensorDial()
|
||||||
|
self.dial.setToolTip("The mouse's rotation on the page since the scan started (the A), its direction "
|
||||||
|
"and speed (the arrow), and other live readings. Tracking is the share of recent "
|
||||||
|
"frames matched by image; the rest relied on the lasers alone (too fast, or lifted).")
|
||||||
self.view = PageView()
|
self.view = PageView()
|
||||||
self.view.selectionChanged.connect(self.on_selection)
|
self.view.selectionChanged.connect(self.on_selection)
|
||||||
for key, slot in ((QKeySequence.Copy, self.view.copy), (QKeySequence.SelectAll, self.view.select_all)):
|
for key, slot in ((QKeySequence.Copy, lambda: self.view.copy()),
|
||||||
|
(QKeySequence.SelectAll, self.view.select_all)):
|
||||||
a = QAction(self)
|
a = QAction(self)
|
||||||
a.setShortcut(key)
|
a.setShortcut(key)
|
||||||
a.triggered.connect(slot)
|
a.triggered.connect(slot)
|
||||||
self.addAction(a)
|
self.addAction(a)
|
||||||
|
|
||||||
self.camera_pane, _ = pane("Camera", self.camera)
|
camera = QWidget()
|
||||||
|
lay = QVBoxLayout(camera)
|
||||||
|
lay.setContentsMargins(0, 0, 0, 0)
|
||||||
|
lay.setSpacing(4)
|
||||||
|
lay.addWidget(self.camera, 1)
|
||||||
|
lay.addWidget(self.dial)
|
||||||
|
self.camera_pane, _ = pane("Camera", camera)
|
||||||
page_pane, self.page_head = pane("Page", self.view)
|
page_pane, self.page_head = pane("Page", self.view)
|
||||||
main = QSplitter(Qt.Horizontal)
|
main = QSplitter(Qt.Horizontal)
|
||||||
for w in (self.camera_pane, page_pane):
|
for w in (self.camera_pane, page_pane):
|
||||||
main.addWidget(w)
|
main.addWidget(w)
|
||||||
main.setSizes([220, 1180])
|
main.setSizes([220, 1180])
|
||||||
|
self.camera_pane.hide()
|
||||||
self.setCentralWidget(main)
|
self.setCentralWidget(main)
|
||||||
self.status_label = QLabel()
|
self.status_label = QLabel()
|
||||||
self.statusBar().addWidget(self.status_label, 1)
|
self.statusBar().addWidget(self.status_label, 1)
|
||||||
@@ -517,13 +722,18 @@ class MainWindow(QMainWindow):
|
|||||||
if state == "scanning" and not self.scanning: # a new scan: clear the old page
|
if state == "scanning" and not self.scanning: # a new scan: clear the old page
|
||||||
self.scanning = True
|
self.scanning = True
|
||||||
self.view.set_lines([])
|
self.view.set_lines([])
|
||||||
|
self.view.angle = 0.0
|
||||||
self.view.set_image(QImage())
|
self.view.set_image(QImage())
|
||||||
self.lines, self.text, self.image, self.path = [], "", None, None
|
self.lines, self.text, self.image, self.path = [], "", None, None
|
||||||
for a in (self.act_save, self.act_ocr, self.act_copy, self.act_savetxt):
|
self.enhanced = None
|
||||||
|
self.page_id += 1
|
||||||
|
for a in (self.act_save, self.act_copy):
|
||||||
a.setEnabled(False)
|
a.setEnabled(False)
|
||||||
self.page_head.setText("Page (scanning)")
|
self.page_head.setText("Page (scanning)")
|
||||||
elif state in ("idle", "done"):
|
elif state in ("idle", "done"):
|
||||||
self.scanning = False
|
self.scanning = False
|
||||||
|
if "sensors" in s and (s["sensors"] or state == "scanning"):
|
||||||
|
self.dial.set_sensors(s["sensors"])
|
||||||
self.act_scan.setText("Stop scan" if state == "scanning" else "Start scan")
|
self.act_scan.setText("Stop scan" if state == "scanning" else "Start scan")
|
||||||
self.act_scan.setEnabled(state in ("idle", "scanning", "done"))
|
self.act_scan.setEnabled(state in ("idle", "scanning", "done"))
|
||||||
parts = [state]
|
parts = [state]
|
||||||
@@ -542,11 +752,11 @@ class MainWindow(QMainWindow):
|
|||||||
|
|
||||||
def on_result(self, path):
|
def on_result(self, path):
|
||||||
self.scanning = False
|
self.scanning = False
|
||||||
self.load(path, auto_ocr=self.act_auto.isChecked())
|
self.load(path, scanned=True)
|
||||||
|
|
||||||
# ---- images ----
|
# ---- images ----
|
||||||
|
|
||||||
def load(self, path, auto_ocr=True):
|
def load(self, path, scanned=False):
|
||||||
try:
|
try:
|
||||||
img = imread(path)
|
img = imread(path)
|
||||||
except OSError:
|
except OSError:
|
||||||
@@ -555,20 +765,51 @@ class MainWindow(QMainWindow):
|
|||||||
log_error(f"cannot read {path}")
|
log_error(f"cannot read {path}")
|
||||||
QMessageBox.warning(self, "myScan", f"cannot read {path}")
|
QMessageBox.warning(self, "myScan", f"cannot read {path}")
|
||||||
return
|
return
|
||||||
self.set_page(img, path)
|
self.set_page(img, path, scanned)
|
||||||
if auto_ocr:
|
|
||||||
self.run_ocr()
|
self.run_ocr()
|
||||||
|
|
||||||
def set_page(self, img, path=None):
|
def set_page(self, img, path=None, scanned=False):
|
||||||
|
"""`scanned`: a fresh stitched page, whose unscanned corners are shown transparent
|
||||||
|
(opened images are shown as they are: their white may be paper)."""
|
||||||
self.image, self.path, self.lines, self.text = img, path, [], ""
|
self.image, self.path, self.lines, self.text = img, path, [], ""
|
||||||
|
self.blank = unscanned(img) if scanned else None
|
||||||
|
self.enhanced = None
|
||||||
|
self.page_id += 1
|
||||||
self.view.set_lines([])
|
self.view.set_lines([])
|
||||||
self.view.set_image(to_qimage(img))
|
self.view.angle = 0.0
|
||||||
|
self.view.set_image(to_qimage(self.shown(), self.blank))
|
||||||
name = os.path.basename(path) if path else "page"
|
name = os.path.basename(path) if path else "page"
|
||||||
self.page_title = f"Page — {name} ({img.shape[1]}×{img.shape[0]})"
|
self.page_title = f"Page — {name} ({img.shape[1]}×{img.shape[0]})"
|
||||||
self.page_head.setText(self.page_title)
|
self.page_head.setText(self.page_title)
|
||||||
self.setWindowTitle(f"myScan — {name}")
|
self.setWindowTitle(f"myScan — {name}")
|
||||||
self.act_save.setEnabled(True)
|
self.act_save.setEnabled(True)
|
||||||
self.act_ocr.setEnabled(True)
|
|
||||||
|
def shown(self):
|
||||||
|
"""The page as shown, saved and read: enhanced or not."""
|
||||||
|
if not self.act_enhance.isChecked() or self.image is None:
|
||||||
|
return self.image
|
||||||
|
if self.enhanced is None:
|
||||||
|
QApplication.setOverrideCursor(Qt.WaitCursor)
|
||||||
|
try:
|
||||||
|
self.enhanced = enhance(self.image, self.blank, self.enhance_strength)
|
||||||
|
finally:
|
||||||
|
QApplication.restoreOverrideCursor()
|
||||||
|
return self.enhanced
|
||||||
|
|
||||||
|
def set_enhance(self, on):
|
||||||
|
if self.image is None:
|
||||||
|
return
|
||||||
|
self.page_id += 1 # read the page again as now shown; drop a reading under way
|
||||||
|
self.view.set_image(to_qimage(self.shown(), self.blank), keep_view=True)
|
||||||
|
self.set_message("page enhanced" if on else "page as scanned")
|
||||||
|
self.run_ocr()
|
||||||
|
|
||||||
|
def set_enhance_strength(self, strength):
|
||||||
|
self.enhance_strength, self.enhanced = strength, None
|
||||||
|
if self.act_enhance.isChecked():
|
||||||
|
self.set_enhance(True)
|
||||||
|
else:
|
||||||
|
self.act_enhance.trigger() # picking a strength turns Enhance on
|
||||||
|
|
||||||
def open_image(self):
|
def open_image(self):
|
||||||
path, _ = QFileDialog.getOpenFileName(self, "Open image", self.default_dir(), IMAGE_FILTER)
|
path, _ = QFileDialog.getOpenFileName(self, "Open image", self.default_dir(), IMAGE_FILTER)
|
||||||
@@ -582,7 +823,10 @@ class MainWindow(QMainWindow):
|
|||||||
path, _ = QFileDialog.getSaveFileName(self, "Save image", base + ".png", IMAGE_FILTER)
|
path, _ = QFileDialog.getSaveFileName(self, "Save image", base + ".png", IMAGE_FILTER)
|
||||||
if path:
|
if path:
|
||||||
try:
|
try:
|
||||||
imwrite(path, self.image)
|
img = self.shown()
|
||||||
|
if self.view.level and abs(self.view.angle) > 0.05:
|
||||||
|
img = level_image(img, self.view.angle)
|
||||||
|
imwrite(path, img)
|
||||||
except OSError as e:
|
except OSError as e:
|
||||||
log_error(f"cannot save {path}: {e}")
|
log_error(f"cannot save {path}: {e}")
|
||||||
QMessageBox.warning(self, "myScan", f"cannot save {path}: {e}")
|
QMessageBox.warning(self, "myScan", f"cannot save {path}: {e}")
|
||||||
@@ -609,28 +853,51 @@ class MainWindow(QMainWindow):
|
|||||||
def run_ocr(self):
|
def run_ocr(self):
|
||||||
if self.image is None or (self.ocr_thread and self.ocr_thread.isRunning()):
|
if self.image is None or (self.ocr_thread and self.ocr_thread.isRunning()):
|
||||||
return
|
return
|
||||||
self.act_ocr.setEnabled(False)
|
|
||||||
self.page_head.setText(f"{self.page_title} · reading…")
|
self.page_head.setText(f"{self.page_title} · reading…")
|
||||||
self.set_message("reading text…")
|
self.set_message("reading text…")
|
||||||
self.ocr_thread = OcrThread(self.ocr, self.image)
|
self.ocr_thread = OcrThread(self.ocr, self.shown())
|
||||||
|
self.ocr_thread.page_id = self.page_id
|
||||||
self.ocr_thread.done.connect(self.on_ocr)
|
self.ocr_thread.done.connect(self.on_ocr)
|
||||||
self.ocr_thread.failed.connect(self.on_ocr_failed)
|
self.ocr_thread.failed.connect(self.on_ocr_failed)
|
||||||
|
self.ocr_thread.finished.connect(self.on_ocr_finished)
|
||||||
self.ocr_thread.start()
|
self.ocr_thread.start()
|
||||||
|
|
||||||
def on_ocr(self, lines, secs):
|
def on_ocr_finished(self):
|
||||||
self.act_ocr.setEnabled(self.image is not None)
|
if self.sender().page_id != self.page_id: # a new page came in while reading the old one
|
||||||
|
self.run_ocr()
|
||||||
|
|
||||||
|
def on_ocr(self, lines, flipped, secs):
|
||||||
|
if self.sender().page_id != self.page_id:
|
||||||
|
return
|
||||||
|
if flipped: # the boxes are on the page turned the right way up
|
||||||
|
self.image = np.ascontiguousarray(self.image[::-1, ::-1])
|
||||||
|
if self.enhanced is not None:
|
||||||
|
self.enhanced = np.ascontiguousarray(self.enhanced[::-1, ::-1])
|
||||||
|
if self.blank is not None:
|
||||||
|
self.blank = self.blank[::-1, ::-1]
|
||||||
|
self.view.set_image(to_qimage(self.shown(), self.blank))
|
||||||
self.lines = lines
|
self.lines = lines
|
||||||
self.text, spans = layout(lines)
|
self.text, spans = layout(lines)
|
||||||
self.view.set_lines(lines, self.text, spans)
|
self.view.set_lines(lines, self.text, spans)
|
||||||
|
d = text_direction(lines)
|
||||||
|
self.view.set_angle(float(np.degrees(np.arctan2(d[1], d[0]))) if lines else 0.0)
|
||||||
conf = f", mean confidence {np.mean([ln.score for ln in lines]):.0%}" if lines else ""
|
conf = f", mean confidence {np.mean([ln.score for ln in lines]):.0%}" if lines else ""
|
||||||
self.page_head.setText(f"{self.page_title} · {len(lines)} lines{conf}")
|
self.page_head.setText(f"{self.page_title} · {len(lines)} lines{conf}")
|
||||||
self.set_message(f"read {len(lines)} lines in {secs:.1f} s" + (" · drag over the text to select it"
|
notes = ["page was upside down, turned"] if flipped else []
|
||||||
if lines else ""))
|
notes += [f"text slanted {self.view.angle:+.1f}°"] if abs(self.view.angle) >= 0.1 else []
|
||||||
|
notes += ["drag over the text to select it"] if lines else []
|
||||||
|
self.set_message(" · ".join([f"read {len(lines)} lines in {secs:.1f} s", *notes]))
|
||||||
self.act_copy.setEnabled(bool(self.text))
|
self.act_copy.setEnabled(bool(self.text))
|
||||||
self.act_savetxt.setEnabled(bool(self.text))
|
|
||||||
|
def set_level(self, on):
|
||||||
|
self.view.set_level(on)
|
||||||
|
if self.lines:
|
||||||
|
what = f"levelled ({-self.view.angle:+.1f}°)" if on else "as stitched"
|
||||||
|
self.set_message(f"showing the page {what}")
|
||||||
|
|
||||||
def on_ocr_failed(self, msg):
|
def on_ocr_failed(self, msg):
|
||||||
self.act_ocr.setEnabled(self.image is not None)
|
if self.sender().page_id != self.page_id:
|
||||||
|
return
|
||||||
self.page_head.setText(self.page_title)
|
self.page_head.setText(self.page_title)
|
||||||
self.set_message(f"OCR failed: {msg}")
|
self.set_message(f"OCR failed: {msg}")
|
||||||
|
|
||||||
@@ -640,21 +907,18 @@ class MainWindow(QMainWindow):
|
|||||||
cb = QGuiApplication.clipboard()
|
cb = QGuiApplication.clipboard()
|
||||||
if cb.supportsSelection(): # X11/Wayland: middle-click pastes it elsewhere
|
if cb.supportsSelection(): # X11/Wayland: middle-click pastes it elsewhere
|
||||||
cb.setText(text, QClipboard.Selection)
|
cb.setText(text, QClipboard.Selection)
|
||||||
preview = " ⏎ ".join(t for t in text.split("\n") if t)
|
preview = " ⏎ ".join(t for t in text.replace("\t", " ").split("\n") if t)
|
||||||
preview = preview if len(preview) <= 90 else preview[:87] + "…"
|
preview = preview if len(preview) <= 90 else preview[:87] + "…"
|
||||||
self.set_message(f"{len(text)} characters selected (Ctrl+C to copy): {preview}")
|
self.set_message(f"{len(text)} characters selected (Ctrl+C to copy): {preview}")
|
||||||
|
|
||||||
def copy_text(self):
|
def view_paragraphs(self, on):
|
||||||
QGuiApplication.clipboard().setText(self.text)
|
self.view.set_paragraphs(on)
|
||||||
self.set_message("text copied to the clipboard")
|
if not self.view.selected_text():
|
||||||
|
self.set_message("copying paragraphs as single lines" if on else "copying lines as on the page")
|
||||||
|
|
||||||
def save_text(self):
|
def copy_text(self):
|
||||||
base = os.path.splitext(self.path)[0] if self.path else os.path.join(self.default_dir(), "page")
|
QGuiApplication.clipboard().setText(self.view.text_range())
|
||||||
path, _ = QFileDialog.getSaveFileName(self, "Save text", base + ".txt", "Text (*.txt)")
|
self.set_message("text copied to the clipboard")
|
||||||
if path:
|
|
||||||
with open(path, "w", encoding="utf-8") as f:
|
|
||||||
f.write(self.text + "\n")
|
|
||||||
self.set_message(f"saved {path}")
|
|
||||||
|
|
||||||
def closeEvent(self, ev):
|
def closeEvent(self, ev):
|
||||||
if self.scan_thread and self.scan_thread.isRunning():
|
if self.scan_thread and self.scan_thread.isRunning():
|
||||||
|
|||||||
@@ -0,0 +1,70 @@
|
|||||||
|
"""Clean up a page for reading: flatten the paper to white, darken the text, and drop
|
||||||
|
bleed-through (the other side's print showing through, fainter than this side's ink).
|
||||||
|
|
||||||
|
The paper's own brightness is estimated per channel by closing the ink away (a max
|
||||||
|
filter wider than a stroke) and smoothing; dividing by it makes the paper 1.0 all over,
|
||||||
|
whatever the lighting or tint. On that scale ink and bleed-through separate by depth:
|
||||||
|
Otsu's threshold over the page splits the ink from everything lighter, and the white
|
||||||
|
point is set part way from it to the paper, so marks lighter than that (the bleed-through,
|
||||||
|
show-through texture, faint shadows) go to white and the ink is stretched to black. Within
|
||||||
|
a few pixels of the ink the white point stays at the paper: faint pixels there are the
|
||||||
|
ink's own soft edges and thin strokes, which would otherwise be eaten away.
|
||||||
|
|
||||||
|
Depth is the darkest channel, so coloured ink is ink too, and colour is kept by scaling
|
||||||
|
what each channel absorbs by one factor: blue ink comes out a deeper blue, grey ink grey.
|
||||||
|
"""
|
||||||
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
|
||||||
|
def paper(img, blank=None, stroke=40, k=4):
|
||||||
|
"""float32 estimate of the paper's colour under each pixel of a BGR or gray uint8 page.
|
||||||
|
`stroke`: a width in pixels larger than any ink stroke; `blank`: a bool mask of pixels
|
||||||
|
that are not page (left out). Done on a k-times smaller copy: the paper varies slowly."""
|
||||||
|
f = img.astype(np.float32)
|
||||||
|
if blank is not None and blank.any():
|
||||||
|
f[blank] = np.median(f[~blank], axis=0) # don't let the unscanned white lift the edges
|
||||||
|
small = cv2.resize(f, None, fx=1 / k, fy=1 / k, interpolation=cv2.INTER_AREA)
|
||||||
|
r = max(1, stroke // k)
|
||||||
|
small = cv2.dilate(small, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2 * r + 1, 2 * r + 1)))
|
||||||
|
small = cv2.GaussianBlur(small, (0, 0), r)
|
||||||
|
bg = cv2.resize(small, (img.shape[1], img.shape[0]), interpolation=cv2.INTER_LINEAR)
|
||||||
|
# Large dark areas (the table beside the page) are not paper: keep them dark.
|
||||||
|
floor = 0.6 * np.percentile(bg if blank is None else bg[~blank], 90, axis=0)
|
||||||
|
return np.maximum(bg, floor.astype(np.float32))
|
||||||
|
|
||||||
|
|
||||||
|
def enhance(img, blank=None, strength=0.25, edge=3):
|
||||||
|
"""The page with its paper flattened to white, ink stretched to black and marks fainter
|
||||||
|
than the ink (bleed-through) whitened. `strength` (0-1): how far the white point moves
|
||||||
|
from the paper towards the ink threshold; `edge`: how many pixels around the ink keep
|
||||||
|
their shading (its soft edges). Returns uint8 of the same shape; `blank` pixels are
|
||||||
|
left as they were."""
|
||||||
|
norm = img.astype(np.float32) / paper(img, blank)
|
||||||
|
# Lightness as the darkest channel, so coloured ink (blue absorbs red) counts as ink.
|
||||||
|
lum = norm.min(axis=2) if norm.ndim == 3 else norm
|
||||||
|
page = lum if blank is None else lum[~blank]
|
||||||
|
hist = np.clip(page * 255, 0, 255).astype(np.uint8)
|
||||||
|
t, _ = cv2.threshold(hist.reshape(-1, 1), 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
|
||||||
|
t /= 255
|
||||||
|
white = 1 - strength * (1 - t)
|
||||||
|
ink = page[page < t]
|
||||||
|
black = np.percentile(ink, 5) if ink.size else 0.0
|
||||||
|
black = min(black, white - 0.05)
|
||||||
|
# Next to ink, faint pixels are its anti-aliased edges and fine strokes, not bleed-through:
|
||||||
|
# there the white point stays at the paper, so strokes keep their full width and shading.
|
||||||
|
core = (lum < t).astype(np.uint8)
|
||||||
|
near = cv2.dilate(core, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2 * edge + 1,) * 2))
|
||||||
|
near = cv2.GaussianBlur(near.astype(np.float32), (0, 0), edge / 2)
|
||||||
|
white = white + (1 - white) * near
|
||||||
|
level = np.clip((lum - black) / (white - black), 0, 1)
|
||||||
|
if norm.ndim == 3:
|
||||||
|
# Scale what each channel absorbs by the same factor, the darkness's stretch: the
|
||||||
|
# ink keeps its hue (its channels' proportions) and gets deeper, never clipped.
|
||||||
|
absorbed = np.clip(1 - norm, 0, None)
|
||||||
|
k = (1 - level) / np.maximum(1 - lum, 1e-3)
|
||||||
|
level = 1 - absorbed * k[..., None]
|
||||||
|
out = (255 * np.clip(level, 0, 1)).round().astype(np.uint8)
|
||||||
|
if blank is not None:
|
||||||
|
out[blank] = img[blank]
|
||||||
|
return out
|
||||||
+81
-4
@@ -4,8 +4,10 @@ LivePreview keeps a downscaled copy of the page: each frame is pasted at its cur
|
|||||||
best pose as it is registered (the last PageGraph solution, extended by odometry), and
|
best pose as it is registered (the last PageGraph solution, extended by odometry), and
|
||||||
when a newer solution arrives the whole preview is redrawn from it, so drift visibly
|
when a newer solution arrives the whole preview is redrawn from it, so drift visibly
|
||||||
snaps into place. LiveSession runs the periodic re-solve and publishes the latest frame,
|
snaps into place. LiveSession runs the periodic re-solve and publishes the latest frame,
|
||||||
the preview and a status dict to a viewer (see webview.py).
|
the preview and a status dict to a viewer (see webview.py), including live sensor readings
|
||||||
|
under "sensors" (see LiveSession.sensors).
|
||||||
"""
|
"""
|
||||||
|
import bisect
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
import traceback
|
import traceback
|
||||||
@@ -13,9 +15,14 @@ import traceback
|
|||||||
import numpy as np
|
import numpy as np
|
||||||
import cv2
|
import cv2
|
||||||
|
|
||||||
|
from .session import bright_level
|
||||||
from .stitching.canvas import Canvas, bbox, normalize, page_orient
|
from .stitching.canvas import Canvas, bbox, normalize, page_orient
|
||||||
from .stitching.filters import feather
|
from .stitching.filters import feather
|
||||||
from .stitching.geometry import corners, pose_matrix
|
from .stitching.geometry import corners, pose_matrix
|
||||||
|
from .stitching.pagegraph import SIGMA_ECC
|
||||||
|
|
||||||
|
MM_PER_PX = 0.0613 # frame pixel on the paper (the EEPROM's pixelSize); about 414 dpi
|
||||||
|
E6_PER_MS = 49.23 # exposure register units per millisecond
|
||||||
|
|
||||||
|
|
||||||
class LivePreview:
|
class LivePreview:
|
||||||
@@ -30,6 +37,7 @@ class LivePreview:
|
|||||||
self.cur = None # pose of the last pasted frame
|
self.cur = None # pose of the last pasted frame
|
||||||
self.seen = None # solution the canvas was drawn from
|
self.seen = None # solution the canvas was drawn from
|
||||||
self.version = 0
|
self.version = 0
|
||||||
|
self.covered = 0 # preview pixels covered by the last render
|
||||||
|
|
||||||
def add(self, corrected):
|
def add(self, corrected):
|
||||||
"""Called after PageGraph.add with the corrected frame it returned. Cheap: pastes
|
"""Called after PageGraph.add with the corrected frame it returned. Cheap: pastes
|
||||||
@@ -78,6 +86,7 @@ class LivePreview:
|
|||||||
return None
|
return None
|
||||||
acc, ws, off, cur = self.c.acc.copy(), self.c.ws.copy(), self.c.off.copy(), self.cur
|
acc, ws, off, cur = self.c.acc.copy(), self.c.ws.copy(), self.c.off.copy(), self.cur
|
||||||
out, covered = normalize(acc, ws, background=background)
|
out, covered = normalize(acc, ws, background=background)
|
||||||
|
self.covered = int(np.count_nonzero(covered))
|
||||||
if out.ndim == 2:
|
if out.ndim == 2:
|
||||||
out = cv2.cvtColor(out, cv2.COLOR_GRAY2BGR)
|
out = cv2.cvtColor(out, cv2.COLOR_GRAY2BGR)
|
||||||
if cur is not None:
|
if cur is not None:
|
||||||
@@ -93,10 +102,15 @@ class LiveSession:
|
|||||||
"""Per-scan live view: re-solves every `every` frames on a background thread and
|
"""Per-scan live view: re-solves every `every` frames on a background thread and
|
||||||
publishes frame, preview and status to `viewer` a few times a second."""
|
publishes frame, preview and status to `viewer` a few times a second."""
|
||||||
|
|
||||||
def __init__(self, graph, viewer, every=45, fps=8):
|
def __init__(self, graph, viewer, every=45, fps=8, mm_per_px=MM_PER_PX):
|
||||||
self.g, self.viewer, self.every, self.period = graph, viewer, every, 1 / fps
|
self.g, self.viewer, self.every, self.period = graph, viewer, every, 1 / fps
|
||||||
|
self.mm = mm_per_px
|
||||||
self.preview = LivePreview(graph)
|
self.preview = LivePreview(graph)
|
||||||
self.raw = None # latest captured frame (set by the capture loop)
|
self.raw = self.rec = None # latest captured frame and its sensor record (on_capture)
|
||||||
|
self.e6, self.times = 0, []
|
||||||
|
self.frame_of = [] # graph frame showing each processed capture (still ones are dropped)
|
||||||
|
self.dist, self.dist_done = 0.0, 0
|
||||||
|
viewer.set_status(sensors=None)
|
||||||
self._solving = True
|
self._solving = True
|
||||||
self._closed = False
|
self._closed = False
|
||||||
self._solver = threading.Thread(target=self._solve_loop, daemon=True)
|
self._solver = threading.Thread(target=self._solve_loop, daemon=True)
|
||||||
@@ -104,9 +118,67 @@ class LiveSession:
|
|||||||
self._solver.start()
|
self._solver.start()
|
||||||
self._render.start()
|
self._render.start()
|
||||||
|
|
||||||
|
def on_capture(self, t, raw, rec, e6):
|
||||||
|
"""Called by the capture loop with each frame, its sensor record and the exposure
|
||||||
|
(E6) it was taken with."""
|
||||||
|
self.times.append(t)
|
||||||
|
self.raw, self.rec, self.e6 = raw, rec, e6
|
||||||
|
|
||||||
def on_frame(self, corrected):
|
def on_frame(self, corrected):
|
||||||
|
"""Called with what PageGraph.add returned for each captured frame, in order."""
|
||||||
|
self.frame_of.append(self.g.n - 1)
|
||||||
|
if corrected is not None:
|
||||||
self.preview.add(corrected)
|
self.preview.add(corrected)
|
||||||
|
|
||||||
|
def sensors(self, window=0.3):
|
||||||
|
"""Live readings for the viewer, or None before the first registered frame:
|
||||||
|
heading: the mouse's rotation on the page since the scan started, in degrees,
|
||||||
|
clockwise as seen in the page preview;
|
||||||
|
velocity: (x, y) mm/s on the page (y down), over the last `window` seconds;
|
||||||
|
speed (mm/s), distance travelled (mm), offset (x, y) from the start (mm),
|
||||||
|
area: page covered so far (cm²);
|
||||||
|
tracking: share of the last 30 frame steps registered by image (the rest
|
||||||
|
only had the lasers, which happens when moving too fast or lifted);
|
||||||
|
lasers: the two laser sensors' surface quality (record bytes 14, 15);
|
||||||
|
exposure (ms), level: paper brightness (0-255), fps."""
|
||||||
|
g, times, rec, frame_of = self.g, self.times, self.rec, self.frame_of
|
||||||
|
k = len(frame_of) - 1 # newest processed capture
|
||||||
|
if k < 1 or rec is None:
|
||||||
|
return None
|
||||||
|
i = frame_of[k] # and the graph frame that shows it
|
||||||
|
c = np.array([g.w / 2, g.h / 2, 1.0])
|
||||||
|
|
||||||
|
def centre(k):
|
||||||
|
# Frame centre in page orientation (page_orient: transpose, then turn 180°),
|
||||||
|
# up to an offset: canvas (x, y) -> page (-y, -x).
|
||||||
|
x, y = (g.pose(k) @ c)[:2]
|
||||||
|
return np.array([-y, -x])
|
||||||
|
M = g.pose(i)
|
||||||
|
# The frame's x axis, (cos θ, sin θ) on the canvas, is (-sin θ, -cos θ) on the page:
|
||||||
|
# it points up at θ = 0 and turns by -θ.
|
||||||
|
heading = -np.degrees(np.arctan2(M[1, 0], M[0, 0]))
|
||||||
|
j = bisect.bisect_left(times, times[k] - window, 0, k)
|
||||||
|
p = centre(i)
|
||||||
|
dt = times[k] - times[j]
|
||||||
|
vel = (p - centre(frame_of[j])) * self.mm / dt if j < k and dt > 0 else np.zeros(2)
|
||||||
|
for _, _, T, _ in g.odo[self.dist_done:i]:
|
||||||
|
self.dist += float(np.hypot(*(T @ c - c)[:2])) * self.mm
|
||||||
|
self.dist_done = i
|
||||||
|
recent = g.odo[max(0, i - 30):i]
|
||||||
|
t = times[-1]
|
||||||
|
fps = (len(times) - 1 - bisect.bisect_left(times, t - 1.0)) / min(1.0, t) if t > 0 else 0.0
|
||||||
|
raw = self.raw
|
||||||
|
return {
|
||||||
|
"heading": float((heading + 180) % 360 - 180),
|
||||||
|
"velocity": [float(v) for v in vel], "speed": float(np.hypot(*vel)),
|
||||||
|
"distance": self.dist, "offset": [float(v) for v in (p - centre(0)) * self.mm],
|
||||||
|
"area": self.preview.covered * (self.mm / self.preview.s) ** 2 / 100,
|
||||||
|
"tracking": sum(e[3] == SIGMA_ECC for e in recent) / len(recent) if recent else 1.0,
|
||||||
|
"lasers": [int(rec.b14), int(rec.b15)],
|
||||||
|
"exposure": self.e6 / E6_PER_MS, "level": bright_level(raw) if raw is not None else 0.0,
|
||||||
|
"fps": float(fps),
|
||||||
|
}
|
||||||
|
|
||||||
def _solve_loop(self):
|
def _solve_loop(self):
|
||||||
"""Re-solve every `every` frames, resting at least as long as the last solve took
|
"""Re-solve every `every` frames, resting at least as long as the last solve took
|
||||||
so it never takes more than about half the CPU from registration."""
|
so it never takes more than about half the CPU from registration."""
|
||||||
@@ -142,7 +214,12 @@ class LiveSession:
|
|||||||
img = self.preview.render()
|
img = self.preview.render()
|
||||||
if img is not None:
|
if img is not None:
|
||||||
self.viewer.publish("page", img)
|
self.viewer.publish("page", img)
|
||||||
self.viewer.set_status(loops=len(self.g.loops))
|
try:
|
||||||
|
sensors = self.sensors()
|
||||||
|
except Exception: # a reading must never stop the preview
|
||||||
|
traceback.print_exc()
|
||||||
|
sensors = None
|
||||||
|
self.viewer.set_status(loops=len(self.g.loops), sensors=sensors)
|
||||||
time.sleep(max(0.0, self.period - (time.time() - t)))
|
time.sleep(max(0.0, self.period - (time.time() - t)))
|
||||||
|
|
||||||
def close(self):
|
def close(self):
|
||||||
|
|||||||
+151
-15
@@ -6,12 +6,16 @@ Latin script (English, Swedish, German, French, ...) as well as Chinese and Japa
|
|||||||
Independent of the scanner hardware, like myscan.stitching.
|
Independent of the scanner hardware, like myscan.stitching.
|
||||||
|
|
||||||
lines = Ocr()(bgr) # [Line(box, text, score, edges)], box = 4x2 corner points
|
lines = Ocr()(bgr) # [Line(box, text, score, edges)], box = 4x2 corner points
|
||||||
|
lines, flipped = Ocr().read(bgr) # ... and whether the page was upside down
|
||||||
text, spans = layout(lines) # plain text; spans[i] = (start, end) of lines[i] in it
|
text, spans = layout(lines) # plain text; spans[i] = (start, end) of lines[i] in it
|
||||||
|
joins = paragraph_joins(lines, text, spans)
|
||||||
|
reflow(text, joins) # the text with paragraphs on one line each
|
||||||
|
|
||||||
Each line knows where its characters are (from the recogniser's per-character boxes),
|
Each line knows where its characters are (from the recogniser's per-character boxes),
|
||||||
so text can be selected on the image: Line.locate maps a point to a character
|
so text can be selected on the image: Line.locate maps a point to a character
|
||||||
position, Line.quad gives the outline of a range of characters.
|
position, Line.quad gives the outline of a range of characters.
|
||||||
"""
|
"""
|
||||||
|
import re
|
||||||
import threading
|
import threading
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
|
||||||
@@ -30,12 +34,9 @@ class Line:
|
|||||||
self.edges = np.linspace(0.0, 1.0, len(self.text) + 1)
|
self.edges = np.linspace(0.0, 1.0, len(self.text) + 1)
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def cy(self):
|
def width(self):
|
||||||
return float(self.box[:, 1].mean())
|
"""Length along the box's top edge."""
|
||||||
|
return float(np.hypot(*(self.box[1] - self.box[0])))
|
||||||
@property
|
|
||||||
def left(self):
|
|
||||||
return float(self.box[:, 0].min())
|
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def height(self):
|
def height(self):
|
||||||
@@ -105,11 +106,25 @@ def char_edges(box, text, chars):
|
|||||||
return np.clip(np.maximum.accumulate(edges), 0.0, 1.0)
|
return np.clip(np.maximum.accumulate(edges), 0.0, 1.0)
|
||||||
|
|
||||||
|
|
||||||
|
def quality(lines):
|
||||||
|
"""Mean confidence per character."""
|
||||||
|
n = sum(len(ln.text) for ln in lines)
|
||||||
|
return sum(ln.score * len(ln.text) for ln in lines) / n if n else 0.0
|
||||||
|
|
||||||
|
|
||||||
class Ocr:
|
class Ocr:
|
||||||
"""Lazily loads the engine (about 0.3 s) on first use; calls are serialised."""
|
"""Lazily loads the engine (about 0.3 s) on first use; calls are serialised.
|
||||||
|
|
||||||
|
RapidOCR's angle classifier, which flips each line it thinks is upside down, is off:
|
||||||
|
it judges lines one by one and now and then turns a perfectly upright line over,
|
||||||
|
reading it as garbage. All lines on a page face the same way, so the page is turned
|
||||||
|
as a whole instead: upright text reads at 0.97-1.0 confidence and upside-down text
|
||||||
|
at 0.55-0.8, so a page reading poorly is read again rotated 180 degrees, and the
|
||||||
|
better reading kept. That also covers pages turned 90 degrees, whose lines are cut
|
||||||
|
out turned one way, which is either right or upside down."""
|
||||||
|
|
||||||
def __init__(self, **params):
|
def __init__(self, **params):
|
||||||
self.params = {"Global.log_level": "warning", **params}
|
self.params = {"Global.log_level": "warning", "Global.use_cls": False, **params}
|
||||||
self._engine, self._lock = None, threading.Lock()
|
self._engine, self._lock = None, threading.Lock()
|
||||||
|
|
||||||
def engine(self):
|
def engine(self):
|
||||||
@@ -119,6 +134,19 @@ class Ocr:
|
|||||||
return self._engine
|
return self._engine
|
||||||
|
|
||||||
def __call__(self, bgr):
|
def __call__(self, bgr):
|
||||||
|
return self.read(bgr)[0]
|
||||||
|
|
||||||
|
def read(self, bgr):
|
||||||
|
"""(lines, flipped): flipped if the page read better rotated 180 degrees, in
|
||||||
|
which case the boxes are in the coordinates of the rotated image."""
|
||||||
|
lines = self.lines(bgr)
|
||||||
|
if quality(lines) < 0.9:
|
||||||
|
turned = self.lines(np.ascontiguousarray(bgr[::-1, ::-1]))
|
||||||
|
if quality(turned) >= max(0.85, quality(lines)):
|
||||||
|
return turned, True
|
||||||
|
return lines, False
|
||||||
|
|
||||||
|
def lines(self, bgr):
|
||||||
with self._lock:
|
with self._lock:
|
||||||
r = self.engine()(bgr, return_word_box=True, return_single_char_box=True)
|
r = self.engine()(bgr, return_word_box=True, return_single_char_box=True)
|
||||||
if r.boxes is None:
|
if r.boxes is None:
|
||||||
@@ -131,18 +159,49 @@ class Ocr:
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def text_direction(lines):
|
||||||
|
"""Unit vector along the text lines: the length-weighted median direction of the
|
||||||
|
lines' top edges (short lines, whose angle is unreliable, only if there is nothing
|
||||||
|
else). A scan held at an angle has every line slanted the same way."""
|
||||||
|
if not lines:
|
||||||
|
return np.array([1.0, 0.0])
|
||||||
|
tops = np.array([ln.box[1] - ln.box[0] for ln in lines])
|
||||||
|
length = np.hypot(tops[:, 0], tops[:, 1])
|
||||||
|
long = length > 3 * np.array([ln.height for ln in lines])
|
||||||
|
if long.any():
|
||||||
|
tops, length = tops[long], length[long]
|
||||||
|
angle = np.arctan2(tops[:, 1], tops[:, 0])
|
||||||
|
order = np.argsort(angle)
|
||||||
|
cum = np.cumsum(length[order])
|
||||||
|
a = angle[order][np.searchsorted(cum, cum[-1] / 2)]
|
||||||
|
return np.array([np.cos(a), np.sin(a)])
|
||||||
|
|
||||||
|
|
||||||
|
def page_coords(lines):
|
||||||
|
"""(along, across) position of each line in the page's own frame, unaffected by the
|
||||||
|
scan's slant: where the line starts along the text direction, and the offset of its
|
||||||
|
centre perpendicular to it (the same anywhere along a straight line)."""
|
||||||
|
d = text_direction(lines)
|
||||||
|
n = np.array([-d[1], d[0]])
|
||||||
|
return [(float((ln.box @ d).min()), float(ln.box.mean(0) @ n)) for ln in lines]
|
||||||
|
|
||||||
|
|
||||||
def rows(lines):
|
def rows(lines):
|
||||||
"""Group lines into visual rows (top to bottom), each sorted left to right. A line
|
"""Group lines into visual rows (top to bottom), each sorted left to right, in the
|
||||||
joins a row when its centre is within half a line height of the row's."""
|
page's frame (page_coords). A line joins a row when its centre is within half a line
|
||||||
|
height of the row's. Returns (rows, the rows' centre offsets)."""
|
||||||
|
pos = {id(ln): p for ln, p in zip(lines, page_coords(lines))}
|
||||||
|
across = lambda ln: pos[id(ln)][1]
|
||||||
out = []
|
out = []
|
||||||
for ln in sorted(lines, key=lambda ln: ln.cy):
|
for ln in sorted(lines, key=across):
|
||||||
for row in out[-3:]:
|
for row in out[-3:]:
|
||||||
if abs(ln.cy - np.mean([r.cy for r in row])) < 0.5 * min(ln.height, row[0].height):
|
if abs(across(ln) - np.mean([across(r) for r in row])) < 0.5 * min(ln.height, row[0].height):
|
||||||
row.append(ln)
|
row.append(ln)
|
||||||
break
|
break
|
||||||
else:
|
else:
|
||||||
out.append([ln])
|
out.append([ln])
|
||||||
return [sorted(row, key=lambda ln: ln.left) for row in out]
|
return ([sorted(row, key=lambda ln: pos[id(ln)][0]) for row in out],
|
||||||
|
[float(np.mean([across(ln) for ln in row])) for row in out])
|
||||||
|
|
||||||
|
|
||||||
def layout(lines):
|
def layout(lines):
|
||||||
@@ -151,8 +210,7 @@ def layout(lines):
|
|||||||
row are separated by a tab. Returns (text, spans), spans[i] = (start, end) of
|
row are separated by a tab. Returns (text, spans), spans[i] = (start, end) of
|
||||||
lines[i] in text."""
|
lines[i] in text."""
|
||||||
index = {id(ln): i for i, ln in enumerate(lines)}
|
index = {id(ln): i for i, ln in enumerate(lines)}
|
||||||
rs = rows(lines)
|
rs, cys = rows(lines)
|
||||||
cys = [float(np.mean([ln.cy for ln in row])) for row in rs]
|
|
||||||
steps = np.diff(cys)
|
steps = np.diff(cys)
|
||||||
pitch = float(np.median(steps)) if len(steps) else 0.0
|
pitch = float(np.median(steps)) if len(steps) else 0.0
|
||||||
spans, parts, pos = [None] * len(lines), [], 0
|
spans, parts, pos = [None] * len(lines), [], 0
|
||||||
@@ -169,3 +227,81 @@ def layout(lines):
|
|||||||
parts.append(ln.text)
|
parts.append(ln.text)
|
||||||
pos += len(ln.text)
|
pos += len(ln.text)
|
||||||
return "".join(parts), spans
|
return "".join(parts), spans
|
||||||
|
|
||||||
|
|
||||||
|
def _extent(ln, o, d):
|
||||||
|
"""(start, end) of a line's box along direction d (unit vector) from point o."""
|
||||||
|
t = (ln.box - o) @ d
|
||||||
|
return float(t.min()), float(t.max())
|
||||||
|
|
||||||
|
|
||||||
|
# a list item's marker: "(a)", "a)", "1.", "iv)", a bullet or dash, then a space
|
||||||
|
MARKER = re.compile(r"(\(?[0-9A-Za-z]{1,3}[.)]|[•·▪◦*\-–])\s+")
|
||||||
|
|
||||||
|
|
||||||
|
def _continues(rs, k):
|
||||||
|
"""Whether the first line of row k + 1 continues the paragraph of the last line of row
|
||||||
|
k: similar text height, the next line isn't indented (except to line up with the
|
||||||
|
text after a list marker, a hanging indent), and line k is full, i.e. the next
|
||||||
|
line's first word would not have fitted in the space left at its end (which works
|
||||||
|
for justified and ragged-right text alike)."""
|
||||||
|
a, b = rs[k][-1], rs[k + 1][0]
|
||||||
|
h = min(a.height, b.height)
|
||||||
|
if not a.text.strip() or not b.text.strip() or abs(a.height - b.height) > 0.3 * max(a.height, b.height):
|
||||||
|
return False
|
||||||
|
o = a.box[0]
|
||||||
|
d = (a.box[1] - a.box[0]) / a.width
|
||||||
|
la, ra = _extent(a, o, d)
|
||||||
|
lb, _ = _extent(b, o, d)
|
||||||
|
if lb > la + 0.8 * h:
|
||||||
|
m = MARKER.match(a.text)
|
||||||
|
if not m or abs(lb - float((a.quad(m.end(), m.end())[0] - o) @ d)) > 0.8 * h:
|
||||||
|
return False
|
||||||
|
left = min(la, lb)
|
||||||
|
right = ra
|
||||||
|
for j in range(max(k - 2, 0), min(k + 4, len(rs))): # the paragraph's right margin
|
||||||
|
c = rs[j][-1]
|
||||||
|
if abs(c.height - h) < 0.3 * h:
|
||||||
|
lj, rj = _extent(c, o, d)
|
||||||
|
if abs(lj - left) < 3 * h:
|
||||||
|
right = max(right, rj)
|
||||||
|
_, e = b.word(0)
|
||||||
|
word = (b.edges[e] - b.edges[0]) * b.width
|
||||||
|
return right - ra < word + 0.5 * h
|
||||||
|
|
||||||
|
|
||||||
|
def paragraph_joins(lines, text, spans):
|
||||||
|
"""Where layout()'s text breaks a paragraph across lines, as edits for reflow():
|
||||||
|
(start, end, replacement) replacing a line break with a space, or removing it along
|
||||||
|
with the hyphen of a word split over two lines (the hyphen stays, without the
|
||||||
|
break, when the next line starts with a capital, as in "Nord-Amerika"). Only a row
|
||||||
|
holding a single line continues a paragraph, and only a single line or a wide last
|
||||||
|
item (as after a separately read section number) runs on into it; rows of several
|
||||||
|
short items are a table."""
|
||||||
|
index = {id(ln): i for i, ln in enumerate(lines)}
|
||||||
|
rs, _ = rows(lines)
|
||||||
|
out = []
|
||||||
|
for k in range(len(rs) - 1):
|
||||||
|
a, b = rs[k][-1], rs[k + 1][0]
|
||||||
|
if len(rs[k + 1]) != 1 or (len(rs[k]) != 1 and a.width < 10 * a.height):
|
||||||
|
continue
|
||||||
|
e = spans[index[id(a)]][1]
|
||||||
|
if text[e:spans[index[id(b)]][0]] != "\n" or not _continues(rs, k):
|
||||||
|
continue
|
||||||
|
if len(a.text) > 1 and a.text[-1] in "-\u00ad¬" and a.text[-2].isalpha():
|
||||||
|
out.append((e - 1, e + 1, "") if b.text[0].islower() else (e, e + 1, ""))
|
||||||
|
else:
|
||||||
|
out.append((e, e + 1, " "))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def reflow(text, joins, a=0, b=None):
|
||||||
|
"""text[a:b] with the paragraph_joins() edits that lie inside it applied."""
|
||||||
|
b = len(text) if b is None else b
|
||||||
|
parts, pos = [], a
|
||||||
|
for s, e, r in joins:
|
||||||
|
if a <= s and e <= b:
|
||||||
|
parts += [text[pos:s], r]
|
||||||
|
pos = e
|
||||||
|
parts.append(text[pos:b])
|
||||||
|
return "".join(parts)
|
||||||
|
|||||||
+9
-6
@@ -17,7 +17,7 @@ from types import SimpleNamespace
|
|||||||
import cv2
|
import cv2
|
||||||
|
|
||||||
from . import calib
|
from . import calib
|
||||||
from .live import LiveSession
|
from .live import MM_PER_PX, LiveSession
|
||||||
from .session import ScanSession
|
from .session import ScanSession
|
||||||
from .stitching import Corrector, PageGraph
|
from .stitching import Corrector, PageGraph
|
||||||
from .xu import XU
|
from .xu import XU
|
||||||
@@ -60,7 +60,7 @@ def wait_press(x, viewer=None, quit=None, poll=0.05):
|
|||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
def scan_page(x, a, frames_dir=None, corrector=None, gains=None, viewer=None):
|
def scan_page(x, a, frames_dir=None, corrector=None, gains=None, viewer=None, mm_per_px=MM_PER_PX):
|
||||||
"""Capture until the button is pressed again. Frames are registered (and saved) on a
|
"""Capture until the button is pressed again. Frames are registered (and saved) on a
|
||||||
worker thread so capture never waits; returns (PageGraph or None, LiveSession or None)."""
|
worker thread so capture never waits; returns (PageGraph or None, LiveSession or None)."""
|
||||||
g, live, rows, work, saves = None, None, [], queue.Queue(), queue.Queue()
|
g, live, rows, work, saves = None, None, [], queue.Queue(), queue.Queue()
|
||||||
@@ -87,14 +87,14 @@ def scan_page(x, a, frames_dir=None, corrector=None, gains=None, viewer=None):
|
|||||||
if g is None:
|
if g is None:
|
||||||
g = PageGraph(im.shape, corrector)
|
g = PageGraph(im.shape, corrector)
|
||||||
if viewer:
|
if viewer:
|
||||||
live = LiveSession(g, viewer)
|
live = LiveSession(g, viewer, mm_per_px=mm_per_px)
|
||||||
th = threading.Thread(target=worker, daemon=True)
|
th = threading.Thread(target=worker, daemon=True)
|
||||||
th.start()
|
th.start()
|
||||||
work.put((n, im, rec.v))
|
work.put((n, im, rec.v))
|
||||||
if sv:
|
if sv:
|
||||||
saves.put((n, im))
|
saves.put((n, im))
|
||||||
if live:
|
if live:
|
||||||
live.raw = im
|
live.on_capture(t, im, rec, s.e6)
|
||||||
if frames_dir:
|
if frames_dir:
|
||||||
rows.append([round(t, 4), n, rec.counter, *rec.v, rec.b14, rec.b15, rec.b16, rec.flags, s.e6,
|
rows.append([round(t, 4), n, rec.counter, *rec.v, rec.b14, rec.b15, rec.b16, rec.flags, s.e6,
|
||||||
round(float(im.mean()), 1), rec.raw.hex()])
|
round(float(im.mean()), 1), rec.raw.hex()])
|
||||||
@@ -112,7 +112,8 @@ def scan_page(x, a, frames_dir=None, corrector=None, gains=None, viewer=None):
|
|||||||
if viewer:
|
if viewer:
|
||||||
viewer.set_status(t=t, frames=n + 1, backlog=work.qsize())
|
viewer.set_status(t=t, frames=n + 1, backlog=work.qsize())
|
||||||
if t - last >= 1:
|
if t - last >= 1:
|
||||||
print(f"\r {t:5.1f}s {n + 1} frames ({g.n} registered) E6={s.e6}", end="", flush=True)
|
print(f"\r {t:5.1f}s {n + 1} frames ({g.n} stitched, {g.stats['still']} still) E6={s.e6}",
|
||||||
|
end="", flush=True)
|
||||||
last = t
|
last = t
|
||||||
print()
|
print()
|
||||||
if viewer:
|
if viewer:
|
||||||
@@ -147,6 +148,7 @@ class Scanner:
|
|||||||
self.corrector = Corrector(None if a.no_flat else calib.FlatField(cal),
|
self.corrector = Corrector(None if a.no_flat else calib.FlatField(cal),
|
||||||
None if a.no_undistort else calib.Undistort(cal))
|
None if a.no_undistort else calib.Undistort(cal))
|
||||||
self.gains = cal["gains_rgb"]
|
self.gains = cal["gains_rgb"]
|
||||||
|
self.mm_per_px = cal["pixel_size_m"] * 1000
|
||||||
|
|
||||||
def wait_press(self, quit=None):
|
def wait_press(self, quit=None):
|
||||||
if self.viewer:
|
if self.viewer:
|
||||||
@@ -163,7 +165,8 @@ class Scanner:
|
|||||||
frames_dir = os.path.join(a.outdir, name) if a.save_frames else None
|
frames_dir = os.path.join(a.outdir, name) if a.save_frames else None
|
||||||
if frames_dir:
|
if frames_dir:
|
||||||
os.makedirs(frames_dir)
|
os.makedirs(frames_dir)
|
||||||
g, live = scan_page(self.x, a, frames_dir, self.corrector, self.gains, viewer)
|
g, live = scan_page(self.x, a, frames_dir, self.corrector, self.gains, viewer,
|
||||||
|
self.mm_per_px)
|
||||||
if live:
|
if live:
|
||||||
live.stop_solving()
|
live.stop_solving()
|
||||||
live.close()
|
live.close()
|
||||||
|
|||||||
@@ -17,6 +17,9 @@ wrong copy. Here every frame keeps its own pose and the poses are solved togethe
|
|||||||
pose; the rest (off the page, lifted, corrupt) only had the odometry and are left out,
|
pose; the rest (off the page, lifted, corrupt) only had the odometry and are left out,
|
||||||
as are frames that disagree with the mosaic of all the others.
|
as are frames that disagree with the mosaic of all the others.
|
||||||
|
|
||||||
|
Frames that show nothing new (the mouse lying still) are dropped on arrival: they cost
|
||||||
|
registration, solving and rendering time and add nothing to the page.
|
||||||
|
|
||||||
A pose M (3x3) maps frame pixels to canvas pixels; an edge (i, j, T) says M_j ~ M_i @ T.
|
A pose M (3x3) maps frame pixels to canvas pixels; an edge (i, j, T) says M_j ~ M_i @ T.
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
@@ -37,6 +40,10 @@ SIGMA_ECC = 1.0 # px, edge registered by ECC
|
|||||||
SIGMA_ODO = 6.0 # px, odometry edge from the motion prior only
|
SIGMA_ODO = 6.0 # px, odometry edge from the motion prior only
|
||||||
SIGMA_LOOP = 1.5 # px, loop closure
|
SIGMA_LOOP = 1.5 # px, loop closure
|
||||||
PRUNE = 5.0 # drop loop edges whose residual exceeds this many sigmas
|
PRUNE = 5.0 # drop loop edges whose residual exceeds this many sigmas
|
||||||
|
STILL_PX = 1.0 # odometry motion (px at the frame corners) below which a frame may be still
|
||||||
|
STILL_DIFF = 0.01 # and thumbnail difference below which it is (sensor noise is ~0.002,
|
||||||
|
# slow motion 0.01-0.03; relative to the mean, so exposure changes don't count)
|
||||||
|
THUMB = (80, 37) # thumbnail size for that test
|
||||||
|
|
||||||
|
|
||||||
class PageGraph:
|
class PageGraph:
|
||||||
@@ -56,10 +63,12 @@ class PageGraph:
|
|||||||
self._hpf = {}
|
self._hpf = {}
|
||||||
self.odo, self.bridges, self.loops = [], [], [] # edges: (i, j, T, sigma)
|
self.odo, self.bridges, self.loops = [], [], [] # edges: (i, j, T, sigma)
|
||||||
self.prev_v = None
|
self.prev_v = None
|
||||||
|
self._thumb = None # of the last frame added
|
||||||
self._feat, self._tried = {}, set() # kept across solves
|
self._feat, self._tried = {}, set() # kept across solves
|
||||||
self._pool = None
|
self._pool = None
|
||||||
self.solution = None # (poses (n, 3), indices of frames to draw)
|
self.solution = None # (poses (n, 3), indices of frames to draw)
|
||||||
self.stats = {"frames": 0, "odo_ecc": 0, "odo_laser": 0, "bridges": 0, "loop_tries": 0, "pruned": 0}
|
self.stats = {"frames": 0, "odo_ecc": 0, "odo_laser": 0, "bridges": 0, "loop_tries": 0, "pruned": 0,
|
||||||
|
"still": 0}
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def n(self):
|
def n(self):
|
||||||
@@ -76,13 +85,29 @@ class PageGraph:
|
|||||||
self._feat[i] = register.detect(self._hp(i))
|
self._feat[i] = register.detect(self._hp(i))
|
||||||
return self._feat[i]
|
return self._feat[i]
|
||||||
|
|
||||||
|
def _still(self, thumb, v):
|
||||||
|
"""Whether a frame with this thumbnail and sensor values shows the same as the last
|
||||||
|
frame added: neither the odometry (without image shift) nor the image moved."""
|
||||||
|
T, _ = self.odometry.predict(self.prev_v, v, self.center)
|
||||||
|
q = np.array([[0, self.w, 0, self.w], [0, 0, self.h, self.h], [1, 1, 1, 1]])
|
||||||
|
if np.abs(T @ q - q).max() >= STILL_PX:
|
||||||
|
return False
|
||||||
|
return cv2.norm(thumb, self._thumb, cv2.NORM_L1) / thumb.size < STILL_DIFF
|
||||||
|
|
||||||
def add(self, frame, v):
|
def add(self, frame, v):
|
||||||
"""frame: HxW or HxWx3 uint8 raw frame; v: its sensor values for the odometry (the
|
"""frame: HxW or HxWx3 uint8 raw frame; v: its sensor values for the odometry (the
|
||||||
myScan record's six raw values). Returns the corrected float frame.
|
myScan record's six raw values). Returns the corrected float frame, or None if the
|
||||||
|
frame was dropped as still (see _still).
|
||||||
Safe to call while solve() runs on another thread."""
|
Safe to call while solve() runs on another thread."""
|
||||||
|
v = np.asarray(v)
|
||||||
|
thumb = cv2.resize(frame, THUMB, interpolation=cv2.INTER_AREA).astype(np.float32)
|
||||||
|
thumb = gray(thumb) / max(float(thumb.mean()), 1.0)
|
||||||
|
if self.n and self._still(thumb, v):
|
||||||
|
self.stats["still"] += 1
|
||||||
|
return None
|
||||||
|
self._thumb = thumb
|
||||||
f = self.correct(frame)
|
f = self.correct(frame)
|
||||||
hs = cv2.resize(highpass_fast(gray(f)), self.half, interpolation=cv2.INTER_AREA)
|
hs = cv2.resize(highpass_fast(gray(f)), self.half, interpolation=cv2.INTER_AREA)
|
||||||
v = np.asarray(v)
|
|
||||||
i = self.n
|
i = self.n
|
||||||
self.frames.append(frame)
|
self.frames.append(frame)
|
||||||
self.hs.append(hs)
|
self.hs.append(hs)
|
||||||
|
|||||||
Reference in New Issue
Block a user