Plan of the UI using from Artosyn — KL730

MT-LB DVE · Periscope board · flex_kneron_ui — Porting & implementation plan

KL730 AArch64 Sony IMX678 FT8201P 10.1" MIPI DSI Camera working UI features in progress
12
Tasks done
5
In progress
22
To do
~174 h
Total remaining
6
Services ported
4
Services to port
Done
In progress
To do
Needs Artosyn port
1
Core Infrastructure — Artosyn Libraries
listener-provider · logger · utility — portable libs ported and integrated
DONE
✓
listener / listener-provider — MQTT backend
Built with -Dlistener_ENABLE_PLATFORM_MQTT=ON and LOCAL_PROVIDER=MQTT. Required CMake quirks: FETCHCONTENT_TRY_FIND_PACKAGE_MODE=NEVER, paho_mqtt_glue forced FetchContent. Explicit -lpaho_mqtt_glue in LDFLAGS. UDP broadcast blocked on Periscope (no eth0). MQTT on lo:1883 works via mosquitto with allow_anonymous true.
Artosyn: artosyn/backends/listener-provider.git
Done 8 h
✓
logger / utility / common-terms — portable header libs
LOG_D/LOG_I/LOG_E macros, ut::result<T>, type aliases (i8/u32/f32…), VERIFY macros. Built for AArch64 with KL730 toolchain. No ARS45 dependency.
Artosyn: general/sw/utility.git, general/sw/logger.git, general/sw/common-terms.git
Done 4 h
✓
mosquitto broker — loopback setup
S20network init script brings up lo (was down at boot on Periscope). Mosquitto config: listener 1883 127.0.0.1, allow_anonymous true. Priority ordering: S20network → S50mosquitto → S99bgs → S98battery → S99flex_kneron_ui.
core/buildroot rootfs-overlay-periscope
Done 4 h
2
universalguilibrary — KL730 Renderer Backend
Created for Artosyn ARS45 — fully re-implemented for KL730 VMF/VDISP
DONE
✓
What universalguilibrary is
universalguilibrary is the GUI abstraction layer originally written for the Artosyn ARS45 platform (used in TTBN6 flexiui). It defines a platform-agnostic rendering interface — GLA_RenderContext (15 function pointers in render_api.h) — that the widget tree calls to draw rectangles, bitmaps, and text, without knowing anything about the underlying hardware. On ARS45 it is backed by the ARS45 HAL (not portable). For KL730 a completely new backend was written from scratch inside universalguilibrary/lib/kl730_renderer/.
Artosyn: universalguilibrary (ARS45 backend — non-portable HAL)
Done —
✓
Kl730Backend — GLA_RenderContext implementation
Implements all GLA_RenderContext hooks for KL730. Key path: end_frame() reads the RGBA working buffer (1920×1200) that the widget tree drew into, converts it to YUV420 using NEON AArch64 intrinsics (BT.601 narrow-range), rotates 90° CW (landscape→portrait 1200×1920 for the FT8201P panel), composites each pixel against the live camera frame (alpha=0 → camera, alpha=255 → GUI), then calls VMF_VDISP_ProcessOneFrame() to push the result to the display via VOC0.
Done 24 h
✓
Canvas zero-copy binding to MemBroker DMA buffer
kl730_backend_init() allocates the RGBA working buffer via MemBroker_GetMemory() (4096-byte aligned, physically contiguous DMA memory). canvas.bind_rgba(kl730_backend_get_rgba_buf(), ...) makes the Canvas write directly into that buffer — the widget tree draws into DMA memory, so end_frame() reads it with no copy. On HOST_MOCK the Canvas uses a plain heap buffer instead.
Done 6 h
✓
Sensor V-flip & 90°CW rotation — software correction
ISP hardware rotation (ISP_MODE_RT) was attempted first but the VMF SSM writer for the _isp_0 pin never initialised — hardware path abandoned. Rotation is done in end_frame() as a transposed scatter-write loop. After that, the IMX678 sensor was delivering a landscape-flipped image (row-mirrored) as a side-effect of the failed ISP_MODE_RT attempt. Fix: camera row index = lr (landscape row, sequential) instead of pc = GUI_H−1−lr (portrait column, reversed). Column unchanged. Both Y and UV planes corrected. Verified working on device.
Done 8 h
3
Buttons General Service (BGS)
F1–F4 hardware buttons → MQTT events → flex_kneron_ui
DONE
✓
BGS — Periscope keycode mapping & MQTT transport
KL730 GPIO driver delivers F1=59, F2=60, F3=61, F4=62 via /dev/input. BGS maps: F1→BUTTON_UP, F2→BUTTON_DOWN, F3→BUTTON_SET, F4→BUTTON_BACK. Build flag: ATN_DEVICE_TYPE=KL730_PERISCOPE → ATN_DEVICE_IS_PERISCOPE=1. No MENU button on hardware — F4/BACK opens menu when menu is closed.
Artosyn: buttons-general-service → KneronVatics/Kneron/buttons-general-service
Done 8 h
✓
flex_kneron_ui — BGS event subscription & queue
lstr::create_local_listener() subscribes to BUTTON_PRESSED / BUTTON_LONG_PRESSED. Thread-safe queue: mutex + vector, swapped atomically each render tick. INIT_READY_BROADCAST sent before render loop to unblock BGS.
Done 4 h
4
Battery General Service
TI BQ27546 fuel gauges + BQ25895 charger — Periscope-specific implementation
~20 h remaining
✓
IBatteryController interface + Bq27546BatteryController
Abstract interface replacing SimpleBatteryController (SARADC-based, absent on Periscope). Bq27546 reads sysfs: /sys/class/power_supply/bq27546-0/ (I2C-0, slot 1), /sys/class/power_supply/bq27546-1/ (I2C-2, slot 2). No raw I2C — kernel driver owns it. Build flag: -DBATTERY_BQ27546=ON.
Artosyn: battery-service → KneronVatics/Kneron/battery-general-service
Done 12 h
◑
On-device verification of BQ27546 sysfs readings
Flash and verify both battery slots read correctly. Confirm capacity, voltage_now, current_now, present, time_to_empty_now. Check MQTT publishes BATTERY_LEVEL {"bat_level1": X, "bat_level2": Y}. Verify InfobarWidget battery icons update in flex_kneron_ui.
In progress 4 h
○
BQ25895 charger INT/CE GPIO handling
BQ25895 per battery: INT (active-low, open-drain) + CE (charge enable, active-low). GPIO 4/5 (bus 0), GPIO 6/7 (bus 2). Subscribe to INT interrupt for charge-state change. Publish EPS_STATUS on MQTT when charge state changes. Guard all GPIO paths with #ifndef BATTERY_BQ27546 (GPIO 31/55 absent on Periscope).
To do 10 h
○
Low-battery shutdown warning in flex_kneron_ui
When either battery reports <5%, show fullscreen warning overlay. At <2%, send SHUTDOWN message to pipeline daemon and initiate safe power-down.
To do 6 h
5
flex_kneron_ui — Core UI (from Artosyn flexiui)
Camera pipeline · kl730_renderer · widget tree · controllers
~36 h remaining
✓
universalguilibrary — kl730_renderer backend
GLA_RenderContext seam for KL730. NEON AArch64 BT.601 RGBA→YUV420 + 90°CW transpose (landscape 1920×1200 → portrait 1200×1920). Alpha composite against camera frame. Sensor V-flip correction: camera row = lr (not pc). VMF_VDISP_ProcessOneFrame → FT8201P.
Artosyn: universalguilibrary (ARS45 backend) → new KL730 backend
Done 24 h
✓
CameraThread — VMF VSRC + DMA_2D double-buffer
IMX678 → IFP 1920×1200 landscape (ISP_MODE_DISABLE). SSM reader on gui_vsrc_ifp_0. DMA_2D copy to MemBroker double-buffer (4096-byte aligned). Atomic lock-free index flip.
Done 16 h
✓
Widget tree — Canvas, MainWidget, InfobarWidget, MainMenuWidget, ReticleWidget
XmlSettings → WidgetFactory → IWidget hierarchy. Canvas RGBA bound zero-copy to MemBroker. XML-driven menu: label IDs, carousel/action/submenu/confirm item types. FreeType text via DejaVuSans. lodepng for PNG icons. LanguageManager EN.xml.
Artosyn: flexiui Canvas, widgets, WidgetsMediator, LanguageManager
Done 32 h
✓
ZoomController + DayNightController
ZoomController: digital zoom 1×/2×/4×/8×/16×, sends ZOOM_SET=<factor> to pipeline. DayNightController: Auto/Day/Night modes, sends DAY_NIGHT_SET=<mode>. Both fully wired through PipelineClient TCP 9001.
Done 8 h
◑
ExposureController — brightness / contrast / sharpness
Sends EXPOSURE_SET=brightness=<v> etc. to pipeline daemon. Pipeline-side ISP parameter mapping to define. Persistence to /mnt/flash/prefs.xml. Menu carousel values need range confirmation with imaging team.
In progress 8 h
◑
RecordingController — start/stop/format SD / snapshot
RECORD_TOGGLE, SNAPSHOT, FORMAT_SD messages to pipeline. Listens for RECORD_STARTED/STOPPED. InfobarWidget REC indicator and timer. SD card present/absent detection from pipeline status.
In progress 10 h
○
SystemController — date/time · language · factory reset
DATE_TIME: launch date -s + hwclock -w, or time-picker widget. LANGUAGE: hot-reload LanguageManager with new EN/RU/etc XML, redraw all widgets. RESTORE_DEFAULTS: wipe /mnt/flash/ (non-flash-mapped files), reboot. RANGE_UNITS: m/yd preference stored in prefs.xml.
Artosyn: LanguageController, TimeController, RetentionController, UnitsController
To do 18 h
6
Artosyn Services — New Ports for KL730
inactivity · display brightness · settings persistence · attitude/IMU
~72 h
◈
inactivity_controller — auto sleep after button inactivity
Port Artosyn inactivity_controller to KL730. Monitors button events (via MQTT) and triggers sleep mode after configurable timeout. On Periscope: no USB detect (GPIO 31 absent), guard with #ifndef BATTERY_BQ27546. Sleep = GPIO16 display-off or DPMS. Wake on any button press. Optional IMU shake-wake if IMU available.
Artosyn: inactivity_controller → TimeoutHandler, AngleHandler
Artosyn port 16 h
◈
display-controller — FT8201P backlight brightness
Port Artosyn display-controller to KL730. Maps user-friendly brightness levels to FT8201P backlight register values (via MIPI DSI command or sysfs if driver exposes it). Persists level to /mnt/flash/prefs.xml. Publishes brightness on MQTT for other services. First step: confirm backlight control path in FT8201P/DW-MIPI-DSI driver.
Artosyn: display-controller → BrightnessMapping, SettingsIntegration
Artosyn port 12 h
◈
settings-controller — NVS or stay with XML+UBIFS — architecture decision
Artosyn uses NVS (JSON/XML + CRC + fallback). KL730 currently uses XML + UBIFS directly in UserSettingsStore. Decision needed: (a) Port settings-controller (NVS + CRC + factory fallback) — more robust, (b) Keep XML+UBIFS — simpler, already working. Recommendation: add CRC verification to UserSettingsStore; avoid full settings-controller port unless NVS-compatible hardware storage is required.
Artosyn: settings-controller → NvsStorageGeneralCore, SerializerBackend
Decision needed 8 h
◈
AttitudeController — IMU pitch/roll display
GUISettings.xml references AttitudeController (SHOW_PITCH_ROLL tab) — not yet implemented. Need to identify IMU chip on Periscope hardware (check schematics in documentation/). Port position-service or read IMU sysfs directly (iio subsystem). PitchRollWidget: horizon line + angle overlay on camera feed. SET_LEVEL = calibrate IMU level reference, persist offset to prefs.xml.
Artosyn: position-service, flexiui AttitudeController, attitude/ widget
Artosyn port 24 h
◈
TrackProjectionController — vehicle track overlay
GUISettings.xml references TrackProjectionController (VEHICLE_TRACKS tab) — not implemented. Source of track data to clarify (CAN bus? external TCP service?). Renders projected track lines as RGBA overlay on Canvas. ALIGN_PROJECTIONS event: manual calibration procedure. This is KL730-specific — no direct Artosyn equivalent.
To do 32 h
7
UI Features & Widgets
InfobarWidget completeness · reticle · recording state · additional overlays
~34 h
○
ReticleController + reticle type selection
GUISettings.xml: add RETICLE_TYPE carousel under image_adj or new tab. ReticleWidget renders current reticle from pre-drawn RGBA icon (lodepng). ReticleController: cycles through types, persists to prefs.xml.
Artosyn: ReticleController, flexiui reticle/ widgets
To do 8 h
○
InfobarWidget — recording timer + SD status
REC indicator blinks when recording. Timer shows elapsed hh:mm:ss. SD card present/absent icon (from pipeline status message). Pipeline publishes RECORD_STARTED {timestamp} → GUI computes elapsed.
To do 6 h
○
Snapshot confirmation feedback widget
On SNAPSHOT_SAVED: brief flash overlay or thumbnail in corner (1–2 s). Received via PipelineClient from pipeline daemon.
To do 4 h
○
Display brightness widget (menu item + slider)
Add DISPLAY_BRIGHTNESS carousel to SETTINGS tab in GUISettings.xml. Depends on display-controller port (Phase 5). Preview: dim/mid/bright/max.
To do 4 h
○
UTF-8 multi-byte decode in FreeType text rendering
Current draw_text() passes raw bytes to FT_Load_Char — non-ASCII (Cyrillic, etc.) renders wrong. Add UTF-8 decode loop (codepoint extraction) before FT_Load_Char. Required for language switching beyond ASCII.
Artosyn: flexiui MultiLanguageTextPrimitive
To do 6 h
○
Dirty-region rendering (skip unchanged widgets)
Currently full-frame Canvas clear + redraw every 33ms. Add is_dirty() flag to IWidget; skip render() call if not dirty. Only composited output pushed to VDISP if canvas changed. ~30% CPU savings estimated for static infobar frames.
Artosyn: flexiui dirty tracking in Canvas render path
To do 6 h
8
Production Readiness & Quality
U-Boot env · SCHED_FIFO · power-off · host mock — pre-ship requirements
~36 h
○
U-Boot env redundancy (CONFIG_ENV_OFFSET_REDUND)
Current config is single-copy — power loss during write corrupts boot settings. Add CONFIG_ENV_OFFSET_REDUND=0x150000, CONFIG_ENV_RANGE=0x40000 to defconfigs. Critical for shipping: loss of day_night_default / zoom_default degrades UX.
To do 6 h
○
SCHED_FIFO for render thread
Set render loop thread to SCHED_FIFO priority to prevent jitter from other processes during compositing. Requires CAP_SYS_NICE or run as root. Test: verify 30fps cadence stable under background system load.
To do 4 h
◑
Power-off via GPIO16 — graceful shutdown daemon
100ms budget — no time for graceful shutdown on power button press. Plan: small C daemon polling GPIO16 interrupt. On event: publish SHUTDOWN on MQTT, wait 500ms for services to write state, then shutdown -h now. Previously deferred — need to revisit priority.
Deferred 12 h
○
HOST_MOCK panel_emulator protocol alignment
panel_emulator currently sends ASCII UDP; BGS sends MQTT JSON. For consistent dev testing, align panel_emulator to send same MQTT button events. Decision deferred until T5-EVO button mapping is settled.
To do 6 h
○
Pre-ship verification checklist
On-device: camera feed + GUI overlay stable 60 min. Memory flat (/proc/<pid>/status VmRSS). MQTT: BGS buttons all respond. Both battery slots read. Charging icon updates on power connect. Menu: all tabs open, carousel items cycle, values persist across reboot. Recording: start/stop, file appears on SD, REC blinks. Snapshot: file saved, feedback shown.
To do 8 h