Fixing the controller register address in DTS
While debugging the HDMI display on the Zedboard--specifically the I2C communication to the ADV7511 chip from the kernel--I noticed that the kernel was also complaining about the OCM device driver, something like| zynq-ocm fffc0000.ps7-ocm: ZYNQ OCM pool: 256 KiB @ 0xe0880000 zynq-ocm fffc0000.ps7-ocm: can't request region for resource [mem 0xfffc0000-0xffffffff] zynq-ocm: probe of fffc0000.ps7-ocm failed with error -16 |
Since the error is there even in the dmesg of log of "successful" reference project, it seems that the problem has been there for some time, but nobody did anything about it. I care about OCM because I am considering it as a primary means of communication with the bare metal application running on CPU1. Later in dmesg, an OCM problem shows up again:
| zynq_pm_remap_ocm: OCM pool is not available zynq_pm_suspend_init: Unable to map OCM. |
The call sequence is zynq_init_late() --> zynq_pm_late_init() --> zynq_pm_suspend_init() --> zynq_pm_remap_ocm() and are called from. zynq_init_late() seems to be part of the platform definition in arch/arm/mach-zynq/common.c:
| DT_MACHINE_START(XILINX_EP107, "Xilinx Zynq Platform") .smp = smp_ops(zynq_smp_ops), .map_io = zynq_map_io, .init_irq = zynq_irq_init, .init_machine = zynq_init_machine, .init_late = zynq_init_late, .init_time = zynq_timer_init, .dt_compat = zynq_dt_match, .reserve = zynq_memory_init, .restart = zynq_system_reset, MACHINE_END |
In the HW exported to SDK (system.hdf), there are 4 memory address ranges:
- OCM controller ps7_ocmc_0: 0xF800C000 - 0xF800CFFF
- SRAM ps7_ram_0: 0x0 - 0x0002FFFF
- SRAM ps7_ram_1: 0xFFFF0000 - 0xFFFFFDFF
- DDR ps7_ddr_0: 0x00100000 - 0x1FFFFFFF; note the bottom 1 MB of the 512 MB DDR is not visible.
0x0003_FFFF) or a high address range (0xFFFC_0000 to 0xFFFF_FFFF) in a granularity of
four independent 64 KB sections via the 4-bit slcr.OCM_CFG[RAM_HI]. The addressing scheme used in this project (adv7511_zed.xpr) seems to be along the line of the example in TRM table 29-5 , but there are important differences:
- There is no SRAM
- Zedboard only has 512 MB DDR3
According to this page, -16 is -EBUSY. The "zynq-ocm" messages are being emitted by <kernel>/arch/arm/mach-zynq/zynq_ocm.c. How does the OCM controller (<kernel>/arch/arm/boot/dts/zynq-zed.dts, which is included by zynq-zed-adv7511.dts) map the memory region 0xFFFC000 - 0xFFFFFFF?
| ps7_ocmc_0: ps7-ocmc@f800c000 { compatible = "xlnx,zynq-ocmc-1.0"; interrupt-parent = <&ps7_scugic_0>; interrupts = <0 3 4>; reg = <0xf800c000 0x1000>; } ; |
But these themselves are inconsistent with /proc/device-tree/amba@0/ps7-ocm@0xfffc0000:
| $ cat /proc/device-tree/amba@0/ps7-ocm@0xfffc0000/name ps7-ocm $ cat /proc/device-tree/amba@0/ps7-ocm@0xfffc0000/compatible xlnx,ps7-ocmc-1.00.axlnx,zynq-ocmc-1.0 $ hexdump /proc/device-tree/amba@0/ps7-ocm@0xfffc0000/interrupts 0000000 0000 0000 0000 0300 0000 0400 000000c $ hexdump /proc/device-tree/amba@0/ps7-ocm@0xfffc0000/reg 0000000 fcff 0000 0400 0000 0000008
The "ZYNQ OCM pool" dmesg is from zynq_ocm_probe() in <kernel>/arch/arm/mach-zynq/zynq_ocm.c:
|
I don't know yet the mapping from the physical to virtual address, but the dev_info line prints only once because the ZYNQ_OCM_BLOCKS (4) number of 64 KB has been concatenated into contiguous 256 KB in an earlier loop. The EBUSY is occurring when device driver calls devm_request_mem_region()--which is #defined to __devm_request_region(dev, &iomem_resource, (start), (n), (name)) in <kernel>/resource.c--with 0xFFFC0000. __devm_request_region can only return NULL for 2 reasons:
- devres_alloc() fails--since this just calls kmalloc in the end, this is unlikely to fail
- __request_region() fails--this probably hits the HW, so is more likely to fail, but not clear why.
With a cold trail, I google for "OCM driver fails to load" and turned up half year old thread. It appears that the failure is due to incorrect OCM *controller* register address passed from the DTB: according to the corrected DTS, it should be <0xf800c000 0x1000>, but the booted target thinks it's <0xfcff0000 0x4000000>, which is CLEARLY incorrect, as the OCM controller register region should only be 64 KB. Even more bizarre, when I reverse compile the generated DTB back into DTS like this, and then look at it, THAT is different than either the DTS source AND what is loaded on the target!
|
The ocm section in reverse compiled DTS appears as follows:
The "compatible" field matches what the target loaded, but where is the "xlnx,ps7-ocmc-1.00.a" coming from? It turns out that ocmc is NOT the only device node whose compatible field gets expanded by the DTS compiler; for example, ps7-xadc@f8007100's compatible field was "xlnx,zynq-xadc-1.00.a"" in DTS, but mysteriously expanded to "xlnx,zynq-xadc-1.00.a", "xlnx,ps7-xadc-1.00.a" in the DTB. The result is the same even if I try running the DTC (device tree compiler) manually and the reverse compile it:
| ps7-ocm@0xfffc0000 { compatible = "xlnx,ps7-ocmc-1.00.a", "xlnx,zynq-ocmc-1.0"; interrupt-parent = <0x1>; interrupts = <0x0 0x3 0x4>; reg = <0xfffc0000 0x40000>; }; |
The "compatible" field matches what the target loaded, but where is the "xlnx,ps7-ocmc-1.00.a" coming from? It turns out that ocmc is NOT the only device node whose compatible field gets expanded by the DTS compiler; for example, ps7-xadc@f8007100's compatible field was "xlnx,zynq-xadc-1.00.a"" in DTS, but mysteriously expanded to "xlnx,zynq-xadc-1.00.a", "xlnx,ps7-xadc-1.00.a" in the DTB. The result is the same even if I try running the DTC (device tree compiler) manually and the reverse compile it:
$ <kernel>/scripts/dtc/dtc -I dtb -O dts <kernel>/arch/arm/boot/dts/zynq-zed-adv7511.dtb > devicetree.dts
|
I finally figured out that I was reading the wrong DTS file: instead of zynq-zed.dts, I should have been reading the zynq.dtsi. I fixed the ocmc entry in <kernel>/arch/arm/boot/dts/zynq.dtsi like this:
ps7_ocmc_0: ps7-ocmc@f800c000 {
compatible = "xlnx,ps7-ocmc-1.00.a", "xlnx,zynq-ocmc-1.0";
interrupt-parent = <&gic>;
interrupts = <0 3 4>;
reg = <0xf800c000 0x1000>;
} ;
When I try this new devicetree.dtb, the initial OCM pool creation error went away!:
compatible = "xlnx,ps7-ocmc-1.00.a", "xlnx,zynq-ocmc-1.0";
interrupt-parent = <&gic>;
interrupts = <0 3 4>;
reg = <0xf800c000 0x1000>;
} ;
Now that the driver has found all OCM in the HW (determined by reading the OCM config register), but mapped the physical address to the kernel virtual address 0xe0880000 (which is constant only for the duration of THAT kernel; if I built another kernel, it may not be the same)--which is NOT the same as the userspace virtual address. The translation table entry itself is created in virt_base = devm_ioremap_resource(&pdev->dev, &zynq_ocm->res[i]); # cat /proc/iomem 00000000-1fdfffff : System RAM 00008000-0062175b : Kernel code 00658000-006b57f3 : Kernel data ... f800c000-f800cfff : /amba@0/ps7-ocmc@f800c000 fffc0000-ffffffff : f800c000.ps7-ocmc |
Using the OCM
As dmesg above shows, the coalesced OCM memory is ioremapped to 0xE0880000 and then added to something called gen_pool (through gen_pool_add_virt). To test writing/reading OCM, I need a simple userspace application--which is a bit complicated when I am cross-compiling (using Buildroot in this case).
Buildroot Eclipse integration
In a previous blog entry, I looked into the Buildroot Eclipse integration, which exposes the Buildroot's cross compile environment in Eclipse. Let's take it out for a spin. Buildroot integration only supports an empty executable project, as you can see below.
If I create a new file main.cpp in the ocm project, I can write a few lines of code:
#include <stdio.h>
int main(int argc, char* argv[]) {
printf("ocm");
return 0;
}
Building this in Eclipse is easy, and runs the correct cross tools to generate an executable ocm:
18:40:34 **** Build of configuration debug for project ocm ****
make all
Building file: ../main.cpp
Invoking: Buildroot ARM C++ Compiler (/mnt/work/zed/buildroot/output)
/mnt/work/zed/buildroot/output/host/usr/bin/arm-buildroot-linux-gnueabi-g++ -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"main.d" -MT"main.d" -o "main.o" "../main.cpp"
Finished building: ../main.cpp
Building target: ocm
Invoking: Buildroot ARM CC Linker (/mnt/work/zed/buildroot/output)
/mnt/work/zed/buildroot/output/host/usr/bin/arm-buildroot-linux-gnueabi-g++ -o "ocm" ./main.o
Finished building target: ocm
18:40:35 Build Finished (took 413ms)
If I just copy this exe to the target's /bin folder, I should be able to run this program. And this is where the value of NFS mounting the rootfs shines: I can just copy the file into the host's NFS export folder!
~/work/zed/workspace$ sudo cp ocm/debug/ocm /export/root/zedbr2/bin/
And when I run this on the target, I see the string "ocm", as expected
# /bin/ocm
ocm
Confirming that a command line program works, I can move onto working with the OCM through mmap system call.
Userspace application to write to OCM
Here is a simple program (requiring sudo) to write 256 KB to the OCM, and then read it back:
#include <iostream>
#include <errno.h>
#include <fcntl.h> //open
#include <unistd.h> //close
#include <sys/mman.h> //mmap
#include <stdio.h>
#include <stdint.h>
using namespace std;
int main(int argc, char* argv[]) {
int err = 0;
uint32_t i;
#define OCM_SIZE 256*1024
#define OCM_LOC 0xFFFC0000//0xe0880000//
printf("ocm: 256 KB @ 0x%x\n", OCM_LOC);
void* ocm = NULL;
uint32_t* buf;
int memf = open("/dev/mem"
, O_RDWR | O_SYNC //do I want cacheing?
);
if(memf < 0) {
err = errno;
cerr << "open " << err;
goto err_;
}
ocm = mmap(NULL, OCM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, memf,
OCM_LOC);
if(ocm == MAP_FAILED) {
err = errno;
cerr << "mmap " << err;
goto err_close;
}
cout << "Writing to OCM" << endl;
for(buf = (uint32_t*)ocm, i=0; i < OCM_SIZE/4; ++i, ++buf)
*buf = i;// << 16;
cout << "Reading OCM" << endl;
for(buf = (uint32_t*)ocm, i=0; i < OCM_SIZE/4; ++i, ++buf) {
if(*buf != i)//(i << 16))
fprintf(stderr, "ocm[%x] = %x\n", i, *buf);
}
err_munmap:
munmap(ocm, OCM_SIZE);
err_close:
if(memf > 0) close(memf);
err_:
return err;
}
#include <iostream>
#include <errno.h>
#include <fcntl.h> //open
#include <unistd.h> //close
#include <sys/mman.h> //mmap
#include <stdio.h>
#include <stdint.h>
using namespace std;
int main(int argc, char* argv[]) {
int err = 0;
uint32_t i;
#define OCM_SIZE 256*1024
#define OCM_LOC 0xFFFC0000//0xe0880000//
printf("ocm: 256 KB @ 0x%x\n", OCM_LOC);
void* ocm = NULL;
uint32_t* buf;
int memf = open("/dev/mem"
, O_RDWR | O_SYNC //do I want cacheing?
);
if(memf < 0) {
err = errno;
cerr << "open " << err;
goto err_;
}
ocm = mmap(NULL, OCM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, memf,
OCM_LOC);
if(ocm == MAP_FAILED) {
err = errno;
cerr << "mmap " << err;
goto err_close;
}
cout << "Writing to OCM" << endl;
for(buf = (uint32_t*)ocm, i=0; i < OCM_SIZE/4; ++i, ++buf)
*buf = i;// << 16;
cout << "Reading OCM" << endl;
for(buf = (uint32_t*)ocm, i=0; i < OCM_SIZE/4; ++i, ++buf) {
if(*buf != i)//(i << 16))
fprintf(stderr, "ocm[%x] = %x\n", i, *buf);
}
err_munmap:
munmap(ocm, OCM_SIZE);
err_close:
if(memf > 0) close(memf);
err_:
return err;
}
When you run it, you should not see any mismatch, but I sometimes see a problem at address 0xFFFCFCF2--only ONLY at that address. Why?
APPENDIX: Buildroot structure to cross compile a userspace application NOT working yet!
Buildroot manual section 9.1 explains the recommended in-tree folders for a board specific packages. Since the OCM test application only makes sense for the Zynq platform, I will comply with the recommended structure. I only need the test application for now, so the following steps are the simplest to get 1 board specific package into buildroot:
First, add too Buildroot's top level Config.in:
menu "Zedboard specific packages"
source "package/avnet/zedboard/Config.in"
endmenu
source "package/avnet/zedboard/ocm/Config.in"
package/avnet/zedboard/ocm/Config.in should offer help for make {x|n|g}config:
config BR2_PACKAGE_ZYNQ_OCM
bool "Zynq OCM user app"
default n
help
Select this if you want a test application for the Zynq OCM
(on-chip memory).
Requires CONFIG_ARCH_ZYNQ in kernel config.
When I run make xconfig in buildroot, I see the correct option under the package category, as you can see below:
~/work/zed/buildroot$ touch package/avnet/zedboard/zedboard.mk
First, add too Buildroot's top level Config.in:
menu "Zedboard specific packages"
source "package/avnet/zedboard/Config.in"
endmenu
This will pull in (my) Zedboard specific packages. Then the board level package/avnet/zedboard/Config.in can pull in specific packages (just 1 for now):
package/avnet/zedboard/ocm/Config.in should offer help for make {x|n|g}config:
config BR2_PACKAGE_ZYNQ_OCM
bool "Zynq OCM user app"
default n
help
Select this if you want a test application for the Zynq OCM
(on-chip memory).
Requires CONFIG_ARCH_ZYNQ in kernel config.
When I run make xconfig in buildroot, I see the correct option under the package category, as you can see below:
~/work/zed/buildroot$ touch package/avnet/zedboard/zedboard.mk
OCM interrupt useless for application
I wondered if I can tell the reader of the OCM of new messages by raising an interrupt, but found in the Zynq TRM section 29.2.5 that the OCM interrupts are only asserted by the HW on parity error or lock request.
