Using the Zynq OCM Linux device driver

Fixing the controller register address in DTS

While debugging the HDMI display on the Zedboard--specifically the I2C communication to the ADV7511 chip from the kernel--I noticed that the kernel was also complaining about the OCM device driver, something like

zynq-ocm fffc0000.ps7-ocm: ZYNQ OCM pool: 256 KiB @ 0xe0880000 zynq-ocm fffc0000.ps7-ocm: can't request region for resource [mem 0xfffc0000-0xffffffff] zynq-ocm: probe of fffc0000.ps7-ocm failed with error -16

Since the error is there even in the dmesg of log of "successful" reference project, it seems that the problem has been there for some time, but nobody did anything about it.  I care about OCM because I am considering it as a primary means of communication with the bare metal application running on CPU1.  Later in dmesg, an OCM problem shows up again:

zynq_pm_remap_ocm: OCM pool is not available zynq_pm_suspend_init: Unable to map OCM.

The call sequence is zynq_init_late() --> zynq_pm_late_init() --> zynq_pm_suspend_init() --> zynq_pm_remap_ocm() and  are called from.  zynq_init_late() seems to be part of the platform definition in arch/arm/mach-zynq/common.c:

DT_MACHINE_START(XILINX_EP107, "Xilinx Zynq Platform")         .smp            = smp_ops(zynq_smp_ops),         .map_io         = zynq_map_io,         .init_irq       = zynq_irq_init,         .init_machine   = zynq_init_machine,         .init_late      = zynq_init_late,         .init_time      = zynq_timer_init,         .dt_compat      = zynq_dt_match,         .reserve        = zynq_memory_init,         .restart        = zynq_system_reset, MACHINE_END

In the HW exported to SDK (system.hdf), there are 4 memory address ranges:

• OCM controller ps7_ocmc_0: 0xF800C000 - 0xF800CFFF
• SRAM ps7_ram_0: 0x0 - 0x0002FFFF
• SRAM ps7_ram_1: 0xFFFF0000 - 0xFFFFFDFF
• DDR ps7_ddr_0: 0x00100000 - 0x1FFFFFFF; note the bottom 1 MB of the 512 MB DDR is not visible.

And according to ug585-Zynq-7000-TRM section 29, Zynq has 256 KB (0x40000) of RAM and 128 KB of ROM (which is not visible to the applications because this is the boot ROM).  The application addressable 256 KB should be mapped to either low range (0x0000_0000 to
0x0003_FFFF) or a high address range (0xFFFC_0000 to 0xFFFF_FFFF) in a granularity of
four independent 64 KB sections via the 4-bit slcr.OCM_CFG[RAM_HI].  The addressing scheme used in this project (adv7511_zed.xpr) seems to be along the line of the example in TRM table 29-5 , but there are important differences:

• There is no SRAM
• Zedboard only has 512 MB DDR3

Note: OCM controller (probably not the OCM itself) starts address 0xFFFC0000 rather than 0xF800C000 defined in system.hdf.  For example, xapp1079 CPU0 application writes to 0xFFFF0000 when writing to the OCM.
According to this page, -16 is -EBUSY.  The "zynq-ocm" messages are being emitted by <kernel>/arch/arm/mach-zynq/zynq_ocm.c.  How does the OCM controller (<kernel>/arch/arm/boot/dts/zynq-zed.dts, which is included by zynq-zed-adv7511.dts) map the memory region 0xFFFC000 - 0xFFFFFFF?

ps7_ocmc_0: ps7-ocmc@f800c000 {   compatible = "xlnx,zynq-ocmc-1.0";   interrupt-parent = <&ps7_scugic_0>;   interrupts = <0 3 4>;   reg = <0xf800c000 0x1000>; } ;

But these themselves are inconsistent with /proc/device-tree/amba@0/ps7-ocm@0xfffc0000:

$ cat /proc/device-tree/amba@0/ps7-ocm@0xfffc0000/name ps7-ocm $ cat /proc/device-tree/amba@0/ps7-ocm@0xfffc0000/compatible  xlnx,ps7-ocmc-1.00.axlnx,zynq-ocmc-1.0 $ hexdump /proc/device-tree/amba@0/ps7-ocm@0xfffc0000/interrupts 0000000 0000 0000 0000 0300 0000 0400 000000c $ hexdump /proc/device-tree/amba@0/ps7-ocm@0xfffc0000/reg 0000000 fcff 0000 0400 0000 0000008 The "ZYNQ OCM pool" dmesg is from zynq_ocm_probe() in <kernel>/arch/arm/mach-zynq/zynq_ocm.c: 152 for (i = 0; i < ZYNQ_OCM_BLOCKS; i++) { 153   unsigned long size; 154   void __iomem *virt_base; 155  156   /* Skip all zero size resources */ 157   if (zynq_ocm->res[i].end == 0) 158     break; ... 174   dev_info(&pdev->dev, "ZYNQ OCM pool: %ld KiB @ 0x%p\n", 175            size / 1024, virt_base); 176 } 177  178 /* Get OCM config space */ 179 res = platform_get_resource(pdev, IORESOURCE_MEM, 0); 180 zynq_ocm->base = devm_ioremap_resource(&pdev->dev, res); 181 if (IS_ERR(zynq_ocm->base)) 182   return PTR_ERR(zynq_ocm->base);

I don't know yet the mapping from the physical to virtual address, but the dev_info line prints only once because the ZYNQ_OCM_BLOCKS (4) number of 64 KB has been concatenated into contiguous 256 KB in an earlier loop.  The EBUSY is occurring when device driver calls devm_request_mem_region()--which is #defined to __devm_request_region(dev, &iomem_resource, (start), (n), (name)) in <kernel>/resource.c--with 0xFFFC0000.  __devm_request_region can only return NULL for 2 reasons:

• devres_alloc() fails--since this just calls kmalloc in the end, this is unlikely to fail
• __request_region() fails--this probably hits the HW, so is more likely to fail, but not clear why.

With a cold trail, I google for "OCM driver fails to load" and turned up half year old thread.  It appears that the failure is due to incorrect OCM *controller* register address passed from the DTB: according to the corrected DTS, it should be <0xf800c000 0x1000>, but the booted target thinks it's <0xfcff0000 0x4000000>, which is CLEARLY incorrect, as the OCM controller register region should only be 64 KB.  Even more bizarre, when I reverse compile the generated DTB back into DTS like this, and then look at it, THAT is different than either the DTS source AND what is loaded on the target!

$ <kernel>/scripts/dtc/dtc -I dtb -O dts <kernel>/arch/arm/boot/dts/zynq-zed-adv7511.dtb  > devicetree.dts

The ocm section in reverse compiled DTS appears as follows:

ps7-ocm@0xfffc0000 {   compatible = "xlnx,ps7-ocmc-1.00.a", "xlnx,zynq-ocmc-1.0";   interrupt-parent = <0x1>;   interrupts = <0x0 0x3 0x4>;   reg = <0xfffc0000 0x40000>; };

The "compatible" field matches what the target loaded, but where is the "xlnx,ps7-ocmc-1.00.a" coming from?  It turns out that ocmc is NOT the only device node whose compatible field gets expanded by the DTS compiler; for example, ps7-xadc@f8007100's compatible field was "xlnx,zynq-xadc-1.00.a"" in DTS, but mysteriously expanded to "xlnx,zynq-xadc-1.00.a", "xlnx,ps7-xadc-1.00.a" in the DTB.  The result is the same even if I try running the DTC (device tree compiler) manually and the reverse compile it:

<kernel>/scripts/dtc$ ./dtc -I dts -O dtb -o ../../arch/arm/boot/dts/zynq-zed-adv7511.dtb ../../arch/arm/boot/dts/zynq-zed-adv7511.dts $ <kernel>/scripts/dtc/dtc -I dtb -O dts <kernel>/arch/arm/boot/dts/zynq-zed-adv7511.dtb  > devicetree.dts

I finally figured out that I was reading the wrong DTS file: instead of zynq-zed.dts, I should have been reading the zynq.dtsi.  I fixed the ocmc entry in <kernel>/arch/arm/boot/dts/zynq.dtsi like this:

ps7_ocmc_0: ps7-ocmc@f800c000 {
  compatible = "xlnx,ps7-ocmc-1.00.a", "xlnx,zynq-ocmc-1.0";
  interrupt-parent = <&gic>;
  interrupts = <0 3 4>;
  reg = <0xf800c000 0x1000>;
} ;

When I try this new devicetree.dtb, the initial OCM pool creation error went away!:

zynq-ocm f800c000.ps7-ocmc: ZYNQ OCM pool: 256 KiB @ 0xe0880000 Now that the driver has found all OCM in the HW (determined by reading the OCM config register), but mapped the physical address to the kernel virtual address 0xe0880000 (which is constant only for the duration of THAT kernel; if I built another kernel, it may not be the same)--which is NOT the same as the userspace virtual address.  The translation table entry itself is created in virt_base = devm_ioremap_resource(&pdev->dev, &zynq_ocm->res[i]); which is where that virtual address printed in dmesg shown above.   I checked the resource flag right before it is IO remapped, and the IORESOURCE_CACHEABLE bit was cleared, so this memory should not be cached at all => no DMB required.  The physical memory address range 0xFFFC0000 ~ 0xFFFFFFFF shows up in /proc because the driver calls gen_pool_add_virt().  You can see the OCM controller registered for the last 256 KB of memory map: # cat /proc/iomem  00000000-1fdfffff : System RAM   00008000-0062175b : Kernel code   00658000-006b57f3 : Kernel data ... f800c000-f800cfff : /amba@0/ps7-ocmc@f800c000 fffc0000-ffffffff : f800c000.ps7-ocmc

Using the OCM

As dmesg above shows, the coalesced OCM memory is ioremapped to 0xE0880000 and then added to something called gen_pool (through gen_pool_add_virt).  To test writing/reading OCM, I need a simple userspace application--which is a bit complicated when I am cross-compiling (using Buildroot in this case).

Buildroot Eclipse integration

In a previous blog entry, I looked into the Buildroot Eclipse integration, which exposes the Buildroot's cross compile environment in Eclipse.  Let's take it out for a spin.  Buildroot integration only supports an empty executable project, as you can see below.

If I create a new file main.cpp in the ocm project, I can write a few lines of code:

#include <stdio.h>

int main(int argc, char* argv[]) {

printf("ocm");

return 0;

}

Building this in Eclipse is easy, and runs the correct cross tools to generate an executable ocm:

18:40:34 **** Build of configuration debug for project ocm ****

make all 

Building file: ../main.cpp

Invoking: Buildroot ARM C++ Compiler (/mnt/work/zed/buildroot/output)

/mnt/work/zed/buildroot/output/host/usr/bin/arm-buildroot-linux-gnueabi-g++ -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"main.d" -MT"main.d" -o "main.o" "../main.cpp"

Finished building: ../main.cpp

Building target: ocm

Invoking: Buildroot ARM CC Linker (/mnt/work/zed/buildroot/output)

/mnt/work/zed/buildroot/output/host/usr/bin/arm-buildroot-linux-gnueabi-g++  -o "ocm"  ./main.o   

Finished building target: ocm

18:40:35 Build Finished (took 413ms)

If I just copy this exe to the target's /bin folder, I should be able to run this program.  And this is where the value of NFS mounting the rootfs shines: I can just copy the file into the host's NFS export folder!

~/work/zed/workspace$ sudo cp ocm/debug/ocm /export/root/zedbr2/bin/

And when I run this on the target, I see the string "ocm", as expected

# /bin/ocm

ocm

Confirming that a command line program works, I can move onto working with the OCM through mmap system call.

Userspace application to write to OCM

Here is a simple program (requiring sudo) to write 256 KB to the OCM, and then read it back:

#include <iostream>
#include <errno.h>
#include <fcntl.h> //open
#include <unistd.h> //close
#include <sys/mman.h> //mmap
#include <stdio.h>
#include <stdint.h>

using namespace std;

int main(int argc, char* argv[]) {
int err = 0;
uint32_t i;
#define OCM_SIZE 256*1024
#define OCM_LOC 0xFFFC0000//0xe0880000//
printf("ocm: 256 KB @ 0x%x\n", OCM_LOC);

void* ocm = NULL;
uint32_t* buf;
int memf = open("/dev/mem"
, O_RDWR | O_SYNC //do I want cacheing?
);
if(memf < 0) {
err = errno;
cerr << "open " << err;
goto err_;
}
ocm = mmap(NULL, OCM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, memf,
OCM_LOC);
if(ocm == MAP_FAILED) {
err = errno;
cerr << "mmap " << err;
goto err_close;
}
cout << "Writing to OCM" << endl;
for(buf = (uint32_t*)ocm, i=0; i < OCM_SIZE/4; ++i, ++buf)
*buf = i;// << 16;
cout << "Reading OCM" << endl;
for(buf = (uint32_t*)ocm, i=0; i < OCM_SIZE/4; ++i, ++buf) {
if(*buf != i)//(i << 16))
fprintf(stderr, "ocm[%x] = %x\n", i, *buf);
}

err_munmap:
munmap(ocm, OCM_SIZE);
err_close:
if(memf > 0) close(memf);
err_:
return err;
}

When you run it, you should not see any mismatch, but I sometimes see a problem at address 0xFFFCFCF2--only ONLY at that address.  Why?

APPENDIX: Buildroot structure to cross compile a userspace application NOT working yet!

Buildroot manual section 9.1 explains the recommended in-tree folders for a board specific packages.  Since the OCM test application only makes sense for the Zynq platform, I will comply with the recommended structure.  I only need the test application for now, so the following steps are the simplest to get 1 board specific package into buildroot:

First, add too Buildroot's top level Config.in:

menu "Zedboard specific packages"
source "package/avnet/zedboard/Config.in"
endmenu

This will pull in (my)  Zedboard specific packages.  Then the board level package/avnet/zedboard/Config.in can pull in specific packages (just 1 for now):

source "package/avnet/zedboard/ocm/Config.in" 

package/avnet/zedboard/ocm/Config.in should offer help for make {x|n|g}config:

config BR2_PACKAGE_ZYNQ_OCM
       bool "Zynq OCM user app"
       default n
       help
         Select this if you want a test application for the Zynq OCM
(on-chip memory).
         Requires CONFIG_ARCH_ZYNQ in kernel config.

When I run make xconfig in buildroot, I see the correct option under the package category, as you can see below:

~/work/zed/buildroot$ touch package/avnet/zedboard/zedboard.mk

OCM interrupt useless for application

I wondered if I can tell the reader of the OCM of new messages by raising an interrupt, but found in the Zynq TRM section 29.2.5 that the OCM interrupts are only asserted by the HW on parity error or lock request.