A simple Linux driver for a camera device

Mirko Damiani
A simple Linux driver
for a camera device
Better Embedded Conference
Firenze, July 8-9, 2013
Terms of interest
●
●
●
GNU/Linux
●
GNU software
●
Linux kernel
Simple driver
●
Easy to develop
●
Easy to use from user-space
Embedded system
●
Custom boards
●
Low resources
2 / 38
Mirko Damiani – [email protected]
Prerequisites
●
●
Entry level
●
GNU/Linux kernel
●
Locking primitives
●
Operating systems
●
Computer architectures
Intermediate level
●
GNU/Linux utilities
●
SW development
3 / 38
Mirko Damiani – [email protected]
Acquisition data path
●
DM368 video process subsystem
●
Camera sensor overview
●
Color filter array
4 / 38
Reference board
●
LeopardBoard 368
●
Texas Instruments DM368
●
ARM926EJ-S
●
432 MHz
●
RAM 128MB
●
NAND 256MB
●
Camera parallel interface
●
Video process subsystem
●
Up to 32K pixels in both
horizontal and vertical direction
5 / 38
Mirko Damiani – [email protected]
Video process subsystem
●
Image Sensor
Interface (ISIF)
●
●
●
●
Get raw data
from sensor
Output to RAM
via a controller
12-to-8 bit
conversion
Paths to other
subsystems
6 / 38
Mirko Damiani – [email protected]
Camera board
●
●
●
High resolution camera
●
2592 x 1944 (5Mp)
●
12 bit pixels @ 96MHz
●
14 fps
●
Color Filter Array (Bayer Filter)
Frame processing
●
Binning, Skipping
●
Black level calibration
Programmable registers
●
I2C interface
7 / 38
Mirko Damiani – [email protected]
Sensor block diagram
●
External clock: 24 MHz
●
Pixel clock: 96 MHz
●
Time needed to sample 1 pixel from the array
Mirko Damiani – [email protected]
8 / 38
Color filter array
●
Bayer format
●
●
●
Invented by Bryce Bayer
at Kodak in 1976
Human eye is more
sensitive to green light
Demosaicing
●
●
Convert raw data into
an RGB image
Take neighborhood pixels
9 / 38
Mirko Damiani – [email protected]
How to read frames
●
Linux drivers
●
System calls: read, mmap
●
Camera parameters
●
Frame rate
10 / 38
Types of Linux drivers
●
●
Character driver
●
A device accessed as a stream of bytes (like a file)
●
Not always possible to move back and forth
●
System calls: open, close, read, write
●
e.g. serial port
Block driver
●
Host a filesystem
●
I/O operations on whole blocks (e.g. 512 bytes)
●
Random access and seeking
●
e.g. hard disk
11 / 38
Mirko Damiani – [email protected]
Accessing a Linux driver
●
●
Filesystem nodes
●
/dev folder
●
mknod <name> [c|b] <major> <minor>
Major and minor numbers
●
Major: identifies the driver associated with a device
–
●
Serial port (major = 4)
Minor: identifies a specific instance of the device
–
COM1 (minor = 64)
–
COM2 (minor = 65)
12 / 38
Mirko Damiani – [email protected]
Camera device driver
●
Character device!
●
How frames are accessed?
●
●
mmap
●
read
How many processes can get frames?
●
●
One process only
How can we apply camera setting?
●
ioctl
●
device parameters
13 / 38
Mirko Damiani – [email protected]
System call: ioctl
●
●
Usage
●
int ioctl(int fd, int request, …)
●
Quite simple
Not useful for rapid prototyping
●
Custom commands and structures must be defined
●
Big switch statement over the command argument
●
Programs must include a common header for definitions
●
Command numbers should be unique over the system
–
Prevent issuing a right command to the wrong device
14 / 38
Mirko Damiani – [email protected]
Sysfs attributes
●
Camera parameters
●
●
●
Exported as files in sysfs
When a file is read or written, sysfs calls the
appropriate show and store callbacks
Sysfs
●
Represents kernel objects
●
Must be compiled with CONFIG_SYSFS
●
mount -t sysfs sysfs /sys
15 / 38
Mirko Damiani – [email protected]
Sysfs attributes
●
●
No special programs needed for configuration
●
echo to write
●
cat to read
●
Available choices can be shown
Limitations
●
Sysfs allocates a PAGE_SIZE buffer (e.g. 4K)
and calls the methods only once
●
No partial read and write operations
●
Values must be simple
–
Avoid structured multi line values
16 / 38
Mirko Damiani – [email protected]
Using camera attributes
●
$> grep "" *
●
●
modes:2592x1944,1296x972,864x648,648x486,432x324
resolution:2592x1944
window:0,0,2592,1944
framesize:5038848
exposure:500
gain:1000
running:0
$> echo 864x648 > resolution
●
sh: write error: Device or resource busy
●
We cannot change resolution when the camera is running
17 / 38
Mirko Damiani – [email protected]
System call: mmap
●
Usage
●
●
●
void *mmap(void *addr, size_t length, int prot, int flags,
int fd, off_t offset)
man mmap
No copy of frames
●
●
●
Buffers are shared between kernel space and user space
User space have to handle a sort of lock/unlock or
queue/dequeue on mmap'ed buffers
Very hard to make it work
18 / 38
Mirko Damiani – [email protected]
System call: read
●
●
Usage
●
ssize_t read(int fd, void *buf, size_t count)
●
Simple and straightforward to use and implement
A copy is required for every read frame
●
copy_to_user
●
Each frame is 4.8MB in size
●
This might lead to performance issues...
●
Anyway it is convenient to try the easy way first
19 / 38
Mirko Damiani – [email protected]
Frame rate at full resolution
●
$> dd if=/dev/cam0 of=/dev/null bs=5038848 count=1
●
●
100 frames
●
●
1+0 records in
1+0 records out
5038848 bytes (4.8MB) copied,
0.387317 seconds, 12.4MB/s
fps = 13.47
10000 frames
●
fps = 13.95
16
14
12
10
fps 8
6
4
2
0
1
10
100
#frames
Mirko Damiani – [email protected]
1000 10000
20 / 38
Driver details
●
Data structure
●
Continuous acquisition
●
Interrupts
●
Memory allocation
●
Driver operations
21 / 38
Frames rotation
●
We hold two list of frames
●
Frames to be written with new data
●
Frames that are ready to be read by user-space
write-list
prev next
Frame A
Frame B
Frame C
prev next
prev next
prev next
read-list
prev next
22 / 38
Mirko Damiani – [email protected]
Frames rotation
●
Frame completed
●
Write data to the write-list head element
●
Append the write-list head element to the tail of read-list
write-list
prev next
read-list
prev next
Frame B
Frame C
prev next
prev next
Frame A
prev next
Mirko Damiani – [email protected]
23 / 38
Frames rotation
●
Frame completed
●
Check whether write-list is empty
●
Take the read-list head
write-list
prev next
read-list
prev next
Frame C
prev next
Frame A
Frame B
prev next
prev next
Mirko Damiani – [email protected]
24 / 38
Frames rotation
●
Frame read by user space
●
Append the read-list head element to the tail of write-list
write-list
prev next
read-list
prev next
Frame C
Frame A
prev next
prev next
Frame B
prev next
Mirko Damiani – [email protected]
25 / 38
Copying to user takes time
●
User pages might not be currently in memory
●
Accessing the swap space
●
Memory overcommit on Linux
●
User space process is put to sleep
write-list
prev next
Frame C
Frame A
prev next
prev next
read-list
prev next
frame-busy
Frame B
prev next
26 / 38
Mirko Damiani – [email protected]
Interrupts
●
At the middle of every frame
●
Take an old buffer and discard its content
●
Switch to the next capture buffer
–
●
Shadow / Event Latched registers
At the end of every frame
●
Make the current buffer available to the user
●
Wake up the sleeping user space process
Frame
Sensor
IRQ1
IRQ2
Mirko Damiani – [email protected]
27 / 38
Protect shared data
●
●
Code that can not sleep
●
There is not a user space process to be put to sleep
●
Interrupt handler
●
Mutexes cannot be used
Spinlocks
●
2 states: locked / unlocked
●
If the lock is available, code goes to the critical section...
●
...otherwise code waits in a tight loop
28 / 38
Mirko Damiani – [email protected]
Request kernel memory
●
We need a contiguous 5MB buffer in memory
●
No more than 4MB
●
Reserve the top of physical RAM
–
Kernel boot parameter: mem=112M
–
dma_declare_coherent_memory
–
dma_mark_declared_memory_occupied
0x00000000
0xC0000000
User space
Kernel space
0GB
3GB
0xC0000000
112MB
0xFFFFFFFF
4GB
0xC8000000
16MB
0xC7000000
Mirko Damiani – [email protected]
29 / 38
Synchronous acquisition
●
●
Camera doesn't run in continuous mode
●
A frame is captured on request
16
●
Camera sensor must be triggered
14
●
One buffer only
12
●
One IRQ only
10
100 frames
●
●
fps = 10.22
10000 frames
●
fps = 10.55
fps
8
6
4
2
0
1
10
100
#frames
Mirko Damiani – [email protected]
1000 10000
30 / 38
Compensation @ 50Hz
●
Neon light is non-uniform during image acquisition
●
Camera shutter must be synchronized
●
Mains frequency is 50Hz in Europe
1
0.8
0.6
I(t)
0.4
0.2
0
0
0.01
0.02
0.03
0.04
0.05
0.06
t
frame
delay
31 / 38
Mirko Damiani – [email protected]
Compensation @ 50Hz
32 / 38
Mirko Damiani – [email protected]
Compensation @ 50Hz
●
●
Camera shutter is continuously triggered
●
Linux kernel high resolution timers
16
●
Average on 2 consecutive frames
14
100 frames
●
●
12
fps = 10.32
10000 frames
●
fps = 10.52
10
fps
8
6
4
2
0
1
10
100
#frames
Mirko Damiani – [email protected]
1000 10000
33 / 38
Driver operations: open
●
Check whether the camera is not already open
●
Check access mode
●
Read-only
●
Clear any previous acquired frames
●
Configure board and camera sensor
●
●
Resolution and window
●
I2C registers
Start acquiring
●
Camera runs in continuous mode
34 / 38
Mirko Damiani – [email protected]
Driver operations: read
●
●
Take least recent frame from read-list
●
Put user space process to sleep, if not available
●
Lock the buffer
Copy its content into the user space buffer
●
●
Honor the “count” parameter of the read syscall
Append the frame to the end of write-list
●
Each read “consumes” one frame
●
No partial reads
35 / 38
Mirko Damiani – [email protected]
Driver operations: close
●
Stop frames acquisition
●
●
Halt camera sensor and DMA transfers
Unregister the current user process
●
Another process is now able to open the camera
36 / 38
Mirko Damiani – [email protected]
Let's Talk
office
+39 055 3984627
e-mail / jabber
[email protected]
web
www.develer.com
License
Creative Commons Attribution 3.0 Unported