camera_pipeline_flow.md

KL730 · Sony IMX678 · MT-LB DVE Periscope · from photons to FT8201P panel + RTSP stream

shared path — sensor → SSM ring buffer

Sony IMX678

3840×2160 STARVIS 2 CMOS · configured via I²C · imx678_3840x2160_ch0.cfg

hardware

MIPI CSI-2

kernel space

IFP — Image Front-end Processor

hardware scale 3840×2160 → 1920×1200 landscape · ISP_MODE_DISABLE (no HW rotation)

hardware

userspace

VMF VSRC · Video Source

opens sensor, manages CSI-2 DMA · writes YUV420 frames into SSM ring on pin gui_vsrc_ifp_0

libvmf

SSM — Shared Segment Memory · ~33 ms/frame

signal splits — two independent VSRC readers

local display path

flex_kneron_ui

CameraThread

blocks on SSM_Reader_ReturnReceiveBuff() · Vector DMA_2D copies Y+U+V into MemBroker double-buffer · atomic index flip after each frame

lock-free double-buffer · m_read_idx

Widget tree render

XmlSettings → WidgetFactory → IWidget::render(Canvas) · writes RGBA into MemBroker buffer · alpha=0 means camera shows through

RGBA working buffer 1920×1200

Kl730Backend::end_frame()

NEON BT.601 RGBA→YUV420 · 90°CW transpose (landscape→portrait) · per-pixel alpha composite with camera · sensor V-flip correction (row=lr)

YUV420 portrait 1200×1920

VMF_VDISP_ProcessOneFrame()

pushes DMA buffer to VOC0 · Video Output Controller

MIPI DSI · dw-mipi-dsi driver

FT8201P 10.1" panel

1200×1920 portrait · 30 fps

encode / stream path

pipeline daemon

VMF VSRC (second handle)

separate VSRC instance · VMF supports multiple simultaneous readers on the same physical sensor

SSM · raw YUV frames

VMF VENC — hardware encoder

H.264 / H.265 · VMF_OVERLAY_CONFIG_T for in-encoder text OSD (zoom level, REC, mode) · zero GUI overhead

SRB — Sync Ring Buffer

rtsp_server

reads SRB · serves RTSP stream over network · SD card recording

IPC with flex_kneron_ui · TCP 9001

← PIPELINE_READY on startup
→ ZOOM_SET / DAY_NIGHT_SET
→ RECORD_TOGGLE / SNAPSHOT
← RECORD_STARTED / SNAPSHOT_SAVED

Network RTSP · SD card

H.264/H.265 encoded stream

Services & processes on device

Process	Init priority	Role
mosquitto	S50	MQTT broker on 127.0.0.1:1883 — IPC bus for all daemons. Requires lo up (S20network) first.
buttons-general-service	S99	reads /dev/input (F1–F4, keycodes 59–62) → publishes BUTTON_PRESSED / BUTTON_LONG_PRESSED on MQTT
battery-general-service	S98	reads BQ27546 sysfs (I²C-0 slot1, I²C-2 slot2) → publishes BATTERY_LEVEL + EPS_STATUS on MQTT
pipeline daemon	S95	VMF VSRC → VENC → SRB → rtsp_server. Listens TCP 9001 for GUI commands.
flex_kneron_ui	S99	GUI render loop (30 fps) + CameraThread + Kl730Backend → local display. Subscribes MQTT for buttons/battery.
S20network	S20	brings up lo, assigns 127.0.0.1/8, adds route 255.255.255.255 → lo for UDP broadcast on loopback-only device

Key platform constraints

MQTT only

Periscope has no eth0 / wlan0 — only loopback. UDP broadcast (INADDR_BROADCAST) returns ENETUNREACH. All daemon IPC uses MQTT over 127.0.0.1:1883.

SW rotate

ISP_MODE_RT (hardware rotation) failed — VMF SSM writer for _isp_0 never initialised. Rotation done in Kl730Backend::end_frame() as NEON transposed scatter-write.

read-only FS

Root filesystem is squashfs (read-only). All runtime state → /mnt/flash/ (UBIFS, SPI-NAND MX35LF4GE4AD). Check /proc/mounts before writing.