一、qemu中物理内存的注册
cpu_register_physical_memory
-->cpu_notify_set_memory
---->kvm_client_set_memory
------>kvm_set_phys_mem
-------->kvm_set_user_memory_region
---------->kvm_vm_ioctl(进入内核)
内核中会调用kvm_vm_ioctl_set_memory_region最终调用到__kvm_set_memory_region函数
在 kvm_set_memory_region函数中有如下代码:
slots->memslots[mem->slot] = new;
old_memslots = kvm->memslots;
rcu_assign_pointer(kvm->memslots, slots);
synchronize_srcu_expedited(&kvm->srcu);
因此函数 kvm_set_memory_region本质是创建并填充了一个临时kvm_memslots结构,并把其赋值给kvm->memslots(全局的)。
二、处理用户态虚拟的地址(主要考虑tlb不能命中的情况)
1、查物理tlb如果不能命中会调用host中do_kvm_tlbmiss
2、do_kvm_tlbmiss会先判断地址是IO地址还是访存的地址,如果是访存地址,会进一步查guest tlb表,如果查guest tlb还没有命中,就会把guest tlb miss异常注入到guest系统中,guest kernel会根据页表来填充guest tlb,当guest调用TLBWI特权指令时,会再次陷入host中,调用do_kvm_cpu异常处理
3、在do_kvm_cpu中模拟TLBWI指令,先填充guest tlb 表项,在调用kvmmips_update_shadow_tlb来更新物理tlb(shadow tlb)
4、在kvmmips_update_shadow_tlb中,通过gfn_to_page和page_to_phys两个函数将gpa转化成hpa,再将hpa填充到物理tlb中
5、gfn_to_page函数(我想讲的重点)这个函数会调用到gfn_to_hva
6、gfn_to_hva调用gfn_to_memslot和gfn_to_hva_memslot
gfn_to_memslot代码如下:
struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn) {
int i;
struct kvm_memslots *slots = kvm_memslots(kvm);
for (i = 0; i < slots->nmemslots; ++i) {
struct kvm_memory_slot *memslot = &slots->memslots[i];
if (gfn >= memslot->base_gfn
&& gfn < memslot->base_gfn + memslot->npages)
return memslot;
}
return NULL;
}
代码中首先调用kvm_memslots获得slots,kvm_memslots代码如下:
static inline struct kvm_memslots *kvm_memslots(struct kvm *kvm) {
return rcu_dereference_check(kvm->memslots,
srcu_read_lock_held(&kvm->srcu)
|| lockdep_is_held(&kvm->slots_lock));
}
本质是return kvm->memsolts。
gfn_to_hva_memslot代码如下:
static unsigned long gfn_to_hva_memslot(struct kvm_memory_slot *slot, gfn_t gfn) {
return slot->userspa
ce_addr + (gfn - slot->base_gfn) * PAGE_SIZE;
}
由此看来gpa到hva的关键是slot->userspace_addr,其在qemu中kvm_set_user_memory_region中通过qemu_safe_ram_ptr函数赋值。
qemu_safe_ram_ptr代码如下:
void *qemu_safe_ram_ptr(ram_addr_t addr) {
RAMBlock *block;
QLIST_FOREACH(block, &ram_list.blocks, next) {
if (addr - block->offset < block->length) {
return block->host + (addr - block->offset);
}
}
fprintf(stderr, "Bad ram offset %" PRIx64 "\n", (uint64_t)addr);
abort();
return NULL;
}
因此得找到block->host,在qemu的qemu_ram_alloc_from_ptr函数中赋值,在该函数中有这么一句话new_block->host = qemu_vmalloc(size);从host系统中分配一个hva地址。
结论:
综合以上分析可以看出,在qemu中调用qemu_ram_alloc主要是分配RAMBlock结构,并将其插入ram_list.blocks链表,它的本质上分配了一个hva地址,把它放到RAMBlock结构host域;调用cpu_register_physical_memory主要填充struct kvm结构的slots域,它的本质是将一个gha地址与hva地址对应起来,将hva放在slot->userspace_addr中,将gha放在slot->base_gfn中。quma通过上面两个函数就把一段gha的空间映射成一段hva空间。
三、console显示过程(基于cirrusfb)
先看一个函数栈:
[<4000000080451164>] cirrusfb_imageblit+0xa0/0x284
[<400000008043ce5c>] bit_putcs+0x3dc/0x48c
[<400000008046eb8c>] do_update_region+0x148/0x1a4
[<40000000804705f4>] update_region+0xb4/0xdc
[<40000000804393bc>] fbcon_switch+0x5b8/0x61c
[<4000000080470ef4>] redraw_screen+0x188/0x2a8
[<4000000080472c84>] take_over_console+0x368/0x3cc
[<4000000080436030>] fbcon_takeover+0x108/0x188
[<4000000080160204>] notifier_call_chain.isra.1+0x40/0x90
[<4000000080160540>] __blocking_notifier_call_chain+0x48/0x68
[<400000008042ee8c>] register_framebuffer+0x2b0/0x2dc
[<400000008010f4b4>] cirrusfb_pci_register+0x608/0x6c4
[<400000008042650c>] pci_device_probe+0x60/0xa0
[<4000000080489008>] driver_probe_device+0x108/0x1f0
[<400000008048915c>] __driver_attach+0x6c/0xa4
[<40000000804879f8>] bus_for_each_dev+0x54/0xa0
[<40000000804881ec>] bus_add_driver+0xf0/0x310
[<4000000080489838>] driver_register+0xe0/0x194
[<4000000080426214>] __pci_register_driver+0x5c/0x11c
[<4000000080886710>] cirrusfb_init+0x164/0x198
[<4000000080870c18>] do_one_initcall+0xbc/0x204
[<4000000080870ecc>] kernel_init+0x16c/0x244
[<40000000801189e8>] kernel_thread_helper+0x10/0x18
从函数栈可以看出,register_framebuffer会触发一个FB_EVENT_FB_REGISTERED事件,调用函数fbcon_fb_registered,该函数中调用fbcon_takeover来接管操作console的函数,从此之后console的操作,就会调用下面函数
static const struct consw fb_con = {
.owner = THIS_MODULE,
.con_startup = fbcon_startup,
.con_init = fbcon_init,
.con_deinit = fbcon_deinit,
.con_clear = fbcon_clear,
.con_putc = fbcon_putc,
.con_putcs = fbcon_putcs,
.con_cursor = fbcon_cursor,
.con_scroll = fbcon_scroll,
.con_bmove = fbcon_bmove,
.con_switch = fbcon_switch,
.con_blank = fbcon_blank,
.con_font_set = fbcon_set_font,
.con_font_get = fbcon_get_font,
.con_font_default = fbcon_set_def_font,
.con_font_copy = fbcon_copy_font,
.con_set_palette = fbcon_set_palette,
.con_scrolldelta = fbcon_scrolldelta,
.con_set_origin = fbcon_set_origin,
.con_invert_region = fbcon_invert_region,
.con_screen_pos = fbcon_screen_pos,
.con_getxy = fbcon_getxy,
.con_resize = fbcon_resize,
.con_debug_enter = fbcon_debug_enter,
.con_debug_leave = fbcon_debug_leave,
};
我们不防以fbcon_putcs函数为例,进一步分析,其代码如下:
static void fbcon_putcs(struct vc_data *vc, const unsigned short *s,int count, int ypos, int xpos) {
struct fb_info *info = registered_fb[con2fb_map[vc->vc_num]];
struct display *p = &fb_display[vc->vc_num];
struct fbcon_ops *ops = info->fbcon_par;
if (!fbcon_is_inactive(vc, info))
ops->putcs(vc, info, s, count, real_y(p, ypos), xpos,
get_color(vc, info, scr_readw(s), 1),
get_color(vc, info, scr_readw(s), 0));
}
它需要调用info->fbcon_par->putcs(info 的数据结构是struct fb_info),info->fbcon_par初始化在函数是fbcon_set_bitops,函数如下:
void fbcon_set_bitops(struct fbcon_ops *ops){
ops->bmove = bit_bmove;
ops->clear = bit_clear;
ops->putcs = bit_putcs;
ops->clear_margins = bit_clear_margins;
ops->cursor = bit_cursor;
ops->update_start = bit_update_start;
ops->rotate_font = NULL;
if (ops->rotate)
fbcon_set_rotate(ops);
}
因此会继续调用bit_putcs,其最终会调用到info->fbops->fb_imageblit(info, image);(info 的数据结构是struct fb_info),info->fbops的初始化函数是cirrusfb_set_fbinfo中,该函数中有info
->fbops = &cirrusfb_ops;一句话,cirrusfb_ops结构如下:
static struct fb_ops cirrusfb_ops = {
.owner = THIS_MODULE,
.fb_open = cirrusfb_open,
.fb_release = cirrusfb_release,
.fb_setcolreg = cirrusfb_setcolreg,
.fb_check_var = cirrusfb_check_var,
.fb_set_par = cirrusfb_set_par,
.fb_pan_display = cirrusfb_pan_display,
.fb_blank = cirrusfb_blank,
.fb_fillrect = cirrusfb_fillrect,
.fb_copyarea = cirrusfb_copyarea,
.fb_sync = cirrusfb_sync,
.fb_imageblit = cirrusfb_imageblit,
};
因此最终会调用到cirrusfb_imageblit函数。
上面一个过程就是一个console写操作,最终调到cirrusfb驱动中cirrusfb_imageblit过程。
四、xserver 显示(基于fbmem)
xserver 下普通的显卡驱动,通常会直接操作寄存器,具体操作就是,先mmap(/dev/mem)io空间的基址,再通过基址加偏移的方式,操作寄存器。
但fbmem是个例外,其不用操作既存器,而是通过ioctl(/dev/fb0),把这些操作丢给内核去做。
但是两者在都没有加速的情况下,framebuffer操作方式是相同的,都是将framebuffer区域通过mmap映射到用户态,然后交给xorg中其他代码处理映射后地址。
五、结论
qemu中cirrus_linear_writeb函数在console下调用很多次而在Xserver不被调用原因如下:
首先在xserver下,我们将framebuffer区域(0x14000000开始的一段区域)mmap成了guest虚拟地址(通过调用/dev/fb0 mmap函数),也就说在xserver所有frambuffer的操作,都是通过这个gva.
其次qema中,在函数map_linear_vram中,通过cpu_register_physical_memory(s->vga.map_addr, s->vga.map_end - s->vga.map_addr, s->vga.vram_offset);(将0x14000000这个gpa和一个hva建立起了联系)
因此在xserver下整个framebuffer操作全部成了内存操作,不是IO操作,过程是gva->gpa->hva->hpa,不会回到qemu中,当然也就不可能访问qemu中的cirrus_linear_writeb函数了。
再次在console下,访console操作最终会调用到cirrusfb_imageblit函数,在cirrusfb_imageblit中有这么一句话memcpy(info->screen_base, image->data, size);其中info->screen_base就是0x14000000remap后(IO空间),因此会回到qemu中,调用cirrus_linear_writeb。
最后,为什么这么关注cirrus_linear_writeb函数,因为在qemu中操作framebuffer表现在往s->vga.vram_ptr中写或从s->vga.vram_ptr读,(s->vga.vram_ptr就是我们说的hva),通过2242 s->vram_offset = qemu_ram_alloc(NULL, "vga.vram", vga_ram_size);
2243 s->vram_ptr = qemu_get_ram_ptr(s->vram_offset);得到。只有在cirrus_linear_writeb函数中,才在往s->vga.vram_ptr这个hva写后,通过cpu_physical_memory_set_dirty将这个区域标记,而在我们更新屏幕是,dirty的区域是我们更新的判断条件。