KL730 UI Rendering — Module Reference

flex_kneron_ui · universalguilibrary kl730_renderer · MT-LB DVE / Periscope board

SoC: Kneron KL730 (AArch64) Panel: FT8201P 10.1" MIPI DSI — 1200×1920 portrait Camera: Sony IMX678 C++14 · Buildroot 2026-07-03

System Architecture

┌─────────────────────────────────────────────────────────────────┐ │ flex_kneron_ui (binary: /atn/flex_kneron_ui) │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Render loop · main thread · 33 ms / ~30 fps │ │ │ │ │ │ │ │ XmlSettings ──► WidgetFactory ──► IWidget tree │ │ │ │ │ render(Canvas) │ │ │ │ BGS events ──► handle_event() ▼ │ │ │ │ (ListenerProvider/MQTT) Canvas (RGBA buf) │ │ │ │ │ bind_rgba() │ │ │ │ ▼ │ │ │ │ Kl730Backend::end_frame() │ │ │ │ RGBA→YUV420 (NEON) │ │ │ │ + 90°CW transpose │ │ │ │ + camera composite │ │ │ │ + VMF_VDISP_ProcessOneFrame │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ ┌───────────────────────────────┐ │ │ │ CameraThread (dedicated thread)│ │ │ │ VMF VSRC → SSM ifp_0 │ │ │ │ Vector DMA_2D → double-buffer │ atomic m_read_idx │ │ │ 1920×1200 landscape YUV420 │◄───────────────────────── │ │ └───────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ VMF VSRC → VOC0 → FT8201P MIPI DSI panel (portrait 1200×1920) BGS (buttons-general-service) / battery-general-service → MQTT lo:1883 Pipeline daemon (TCP 9001) → zoom · day/night · recording commands

Modules

Module Source file(s) Role
main.cpp app/src/main.cpp Entry point. Loads XML config, connects to pipeline daemon (TCP 9001), builds widget tree, initialises KL730 backend + camera thread, subscribes to BGS/battery events via ListenerProvider (MQTT 127.0.0.1:1883), runs render loop at 30 fps. Handles SIGINT/SIGTERM for clean shutdown.
XmlSettings app/src/utils/XmlSettings.cpp
app/inc/utils/XmlSettings.hpp		 Parses GUISettings.xml via pugixml. Resolves config path: /mnt/flash/GUISettings.xml (runtime override) → /atn/GUISettings.xml (factory). Produces canvas dimensions, controller list, and widget tree specification.
WidgetFactory app/src/utils/WidgetFactory.cpp Reads widget spec from XmlSettings. Constructs all IWidget instances (MainWidget, InfobarWidget, MainMenuWidget, ReticleWidget). Wires widgets to their controllers. Returns the root widget pointer.
IWidget app/inc/widgets/IWidget.hpp Abstract widget interface: render(Canvas&) and handle_event(Event). Concrete implementations: MainWidget (root compositor), InfobarWidget (status bar — battery, EPS, mode indicators), MainMenuWidget (F1–F4 navigation), ReticleWidget (aiming reticle overlay).
Canvas app/src/utils/Canvas.cpp
app/inc/utils/Canvas.hpp		 RGBA draw surface (1920×1200). On device, bound directly to Kl730Backend's MemBroker DMA buffer via bind_rgba() — zero-copy. Exposes fill, draw_rect, draw_text, draw_icon, draw_png. On HOST_MOCK, owns its own heap buffer.
WidgetsMediator app/src/utils/WidgetsMediator.cpp Observer/mediator pattern for widget-to-widget notifications (e.g., menu open/close toggling InfobarWidget visibility). Decouples widget interdependencies.
PipelineClient app/src/utils/PipelineClient.cpp TCP client to the pipeline daemon on port 9001. Sends zoom / day-night / recording commands; receives the pipeline-ready handshake at startup. Blocks with 5-second timeout on wait_pipeline_ready().
DayNightController app/src/controllers/DayNightController.cpp Tracks day/night mode state. On mode change: sends command to pipeline daemon via PipelineClient, notifies widgets via WidgetsMediator.
ZoomController app/src/controllers/ZoomController.cpp Manages digital zoom level. Sends zoom step command to pipeline daemon on button events.
ExposureController app/src/controllers/ExposureController.cpp Manages CMOS sensor exposure settings. Interacts with pipeline daemon for exposure adjustments.
RecordingController app/src/controllers/RecordingController.cpp Start/stop recording control. Sends recording commands to pipeline daemon; updates recording-state indicator in widgets.
Icon app/src/utils/Icon.cpp
app/inc/utils/Icon.hpp		 PNG icon loader using lodepng. Caches decoded RGBA icon bitmaps in memory. Used by Canvas::draw_icon(). Icons installed at /atn/resources/icons/.
LanguageManager app/src/utils/LanguageManager.cpp Loads language XML files (e.g., /atn/languages/EN.xml) for localised UI strings. Provides string lookup by key.
UserSettingsStore app/src/utils/UserSettingsStore.cpp Persists user preferences to /mnt/flash/ (UBIFS, SPI-NAND). Guards all writes with /proc/mounts check — returns error if flash is not mounted.
CameraThread app/src/utils/CameraThread.cpp
app/inc/utils/CameraThread.hpp		 Dedicated thread owning the VMF VSRC camera pipeline. Opens IMX678 sensor via imx678_3840x2160_ch0.cfg; IFP scales to landscape 1920×1200 (ISP_MODE_DISABLE). Blocking SSM read on gui_vsrc_ifp_0. Hardware DMA_2D copy to MemBroker double-buffer (4096-byte aligned). Atomically flips m_read_idx after each frame. Buffer constants: c_out_w=1920, c_out_h=1200, c_out_stride=1984.
Kl730Backend universalguilibrary/lib/kl730_renderer/src/Kl730Backend.cpp
universalguilibrary/lib/kl730_renderer/inc/Kl730Backend.hpp		 Platform seam implementing the GLA_RenderContext interface (render_api.h). end_frame(): NEON-accelerated BT.601 RGBA→YUV420 conversion with 90°CW transpose (landscape→portrait), per-pixel camera composite (alpha-blend + sensor V-flip correction), then VMF_VDISP_ProcessOneFrame(). Allocates portrait DMA double-buffer (PORT_W=1200, PORT_H=1920, PORT_STRIDE=1216) via MemBroker. Exposes lifecycle API: kl730_backend_init(), kl730_backend_destroy(), kl730_backend_set_camera(), kl730_backend_get_rgba_buf().
ListenerProvider (BGS) external: liblistener_provider_lib UDP/local MQTT subscriber. Receives: BUTTON_PRESSED / BUTTON_LONG_PRESSED (F1=Up, F2=Down, F3=Set, F4=Back, keycodes 59–62 from BGS), BATTERY_LEVEL (JSON: bat_level1/bat_level2 from TI BQ27546 fuel gauges on I2C-0/I2C-2), EPS_STATUS (BQ25895 charger — 1=charging). Callbacks push to a mutex-protected queue; render loop drains each tick.
lodepng third_party/lodepng/lodepng.cpp Single-file PNG decode library. Used exclusively by the Icon module to decode PNG icons into RGBA bitmaps at load time.
HOST_MOCK (dev only) mock/src/DisplaySDL.cpp
mock/src/ButtonServer.cpp		 x86 development build only (compiled with -DHOST_MOCK). SDL2 window renders the Canvas RGBA buffer at native size. ButtonServer listens on UDP 9002 for panel_emulator button injection. Not shipped on device.

Per-Frame Render Pipeline

1
Boot & Initialisation
main() loads GUISettings.xml → XmlSettings. Connects to pipeline daemon (TCP 9001, 5 s timeout). WidgetFactory builds the IWidget tree. kl730_backend_init() allocates RGBA working buffer (1920×1200×4 = ~9 MB heap) and portrait YUV420 DMA double-buffer (1216×1920×1.5 ≈ 3.5 MB MemBroker each slot). Canvas is bound to the RGBA buffer — zero copy from this point.
2
CameraThread start (background)
CameraThread::start("gui_vsrc", "/kneron/Resource/VIC/0/imx678_3840x2160_ch0.cfg"). VMF VSRC opens sensor, IFP downscales 3840×2160 → 1920×1200 landscape (ISP_MODE_DISABLE). SSM reader opens gui_vsrc_ifp_0. DMA_2D descriptor configured (Y + Cb + Cr planes, stride 1984). Frame loop: SSM receive → DMA_2D copy to write slot → atomic index flip. Render thread reads latest slot non-blocking via get_frame().
3
Event drain (each tick, ~33 ms)
ListenerProvider callbacks push Event structs to btn_queue (mutex-protected). Render loop swaps the queue atomically, then calls root→handle_event(ev) for each:
  • ButtonPress → menu navigation, zoom, mode change
  • BatteryLevel → InfobarWidget updates battery icons (BQ27546 × 2)
  • EpsStatus → InfobarWidget updates charging icon (BQ25895)
4
Widget render → Canvas
canvas.fill(Color::transparent()) — zeroes the RGBA buffer (all alpha = 0, camera shows through by default).
root→render(canvas) — widget tree draws into the RGBA buffer: InfobarWidget paints status icons, MainMenuWidget paints menu (if open), ReticleWidget paints reticle crosshair, MainWidget composites children. Alpha = 0 pixels remain transparent → camera behind. Alpha = 255 pixels are opaque GUI → occludes camera.
5
Kl730Backend::end_frame() — NEON compose
Reads m_read_idx atomically → gets latest camera Y/U/V plane pointers and strides.
Y plane — landscape-row-major loop for sequential RGBA and camera reads:
  • Outer loop: lr = 0..1199 (landscape row) → portrait col pc = 1199 − lr
  • Camera row: cam_y + lr × cam_stride (sensor V-flip correction: row = lr not pc)
  • NEON inner: vld4q_u8 loads 16 RGBA pixels; vld1q_u8 loads 16 cam_y
  • Y = (66R + 129G + 25B + 4224) >> 8 — BT.601 narrow-range, result ∈ [16, 235]
  • Blend: (Y×α + cam_y×(255−α)) >> 8; vbsl overrides: α=0 → cam, α=255 → GUI
  • Scatter write: dst_y[lc × PORT_STRIDE + pc] (90°CW: portrait row = lc)
UV plane — 4:2:0 downsampled, even rows/cols:
  • Camera UV row = lr/2 (same V-flip correction), col = lc/2
  • Alpha-blend or override using top-left pixel of each 2×2 block
VMF_VDISP_ProcessOneFrame() → VOC0 DMA → FT8201P MIPI DSI.

Buffer Geometry

BufferDimensionsStrideAllocator
RGBA working buffer (GUI)1920 × 1200 × 4 B7680 B/rowheap (new u8[])
Camera landscape Y1984 × 12001984 B/rowMemBroker (DMA-mapped)
Camera landscape UV992 × 600 each992 B/rowMemBroker (DMA-mapped)
Portrait display Y1216 × 19201216 B/rowMemBroker (4096-aligned)
Portrait display UV608 × 960 each608 B/rowMemBroker (4096-aligned)
Coordinate mapping (90°CW landscape→portrait): landscape pixel (lc, lr) → portrait position dst_y[lc × PORT_STRIDE + (GUI_H−1−lr)]. Portrait row = landscape column; portrait column = (GUI_H−1) − landscape row.

Key Design Decisions

No ARS45 backend
The GLA_RenderContext seam (render_api.h) from universalguilibrary is used, but the ARS45 HAL backend from the Artosyn TTBN-6 project is not portable to KL730. A new kl730_renderer backend was implemented from scratch targeting VMF VSRC + VMF VDISP.
Software 90°CW rotation
ISP hardware rotation (ISP_MODE_RT) was attempted but failed — the SSM writer for _isp_0 never initialised. The rotation is done in end_frame() as a transposed write loop. NEON acceleration brings the Y-plane cost to ~1 ms for 1920×1200.
Lock-free camera double-buffer
CameraThread writes to slot 1−m_read_idx and flips m_read_idx atomically after each DMA_2D completes. Render thread reads the current slot without any mutex — zero stall on the 30 fps render thread.
Canvas zero-copy binding
canvas.bind_rgba(kl730_backend_get_rgba_buf(), ...) gives Canvas direct access to the MemBroker DMA buffer. Widget draw calls write directly to composited memory — no intermediate buffer copy before end_frame().
Sensor V-flip correction (software)
The failed ISP_MODE_RT attempt left the sensor with its landscape rows mirrored. Correction in the compose loop: camera Y row = lr (landscape row iteration variable) instead of pc (GUI_H−1−lr). Column unchanged. UV: row = lr/2.
MQTT on loopback only
Periscope has no Ethernet or Wi-Fi — only the lo interface (127.0.0.1). BGS and battery-general-service communicate via mosquitto on lo:1883. S20network init script must run before mosquitto.
Read-only rootfs / UBIFS for state
Runtime rootfs is read-only (squashfs). All writable state (user settings, config overrides) goes to /mnt/flash/ (UBIFS, SPI-NAND MX35LF4GE4AD). UserSettingsStore verifies /proc/mounts before writing.
Pipeline daemon separation
The camera/encode pipeline runs as a separate daemon process. flex_kneron_ui is GUI-only — it connects to the daemon via TCP 9001 (PipelineClient) to send zoom/mode/recording commands and receive the ready signal.

IPC & External Dependencies

ChannelTransportDirectionMessages
Pipeline daemonTCP 127.0.0.1:9001BidirectionalPIPELINE_READY, ZOOM_SET, DAYNIGHT_SET, RECORDING_START, RECORDING_STOP
BGS (buttons)MQTT lo:1883 (ListenerProvider)SubscribeBUTTON_PRESSED, BUTTON_LONG_PRESSED → F1/F2/F3/F4 (Up/Down/Set/Back)
Battery serviceMQTT lo:1883 (ListenerProvider)SubscribeBATTERY_LEVEL {bat_level1, bat_level2}, EPS_STATUS {0|1}
VMF VSRCSSM shared memorySubscribe (camera frames)YUV420 frames on pin gui_vsrc_ifp_0, 1920×1200, ~30 fps
VMF VDISPVMF internal (VOC0)Push (display output)YUV420 portrait frames 1200×1920 → FT8201P MIPI DSI panel