Account NvMap allocated memory into both RSS and CG tracking to make
efficient OOM kill decisions during memory pressure.
NvMap allocates memory via kernel APIs like alloc_pages, the kernel
memory is not accounted on behalf of process who requests the
allocation. Hence in case OOM, the OOM killer never kills the process
who has allocated memory via NvMap even though this process might be
holding most of the memory.
Solve this issue using following approach:
- Use __GFP_ACCOUNT and __GFP_NORETRY flag
- __GFP_NORETRY will not let the current allocation flow to go into OOM
path, so that it will never trigger OOM.
- __GFP_ACCOUNT causes the allocation to be accounted to kmemcg. So any
allocation done by NvMap will be definitely accounted to kmemcg and
cgroups can be used to define memory limits.
- Add RSS counting for the process which allocates by NvMap, so that OOM
score for that process will get updated and OOM killer can pick this
process based upon the OOM score.
- Every process that has a reference to NvMap Handle would have the
memory size accounted into its RSS. On releasing the reference to
handle, the RSS would be reduced.
Bug 5222690
Change-Id: I3fa9b76ec9fc8d7f805111cb96e11e2ab1db42ce
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3447072
(cherry picked from commit 858d73775a)
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3458546
GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>
Tested-by: Amulya Yarlagadda <ayarlagadda@nvidia.com>
Reviewed-by: Amulya Yarlagadda <ayarlagadda@nvidia.com>
This reverts commit b90900fdece2eb002b5061681aead4a74401b484.
After a deeper analysis, it was found that multithreaded cache flush is
not required. Only gathering the pages and then doing cache flush is
enough to get the perf improvement, the cache flush time of each page is
almost half in this case as compared with not gathering the pages. So it
means there is continuous cache eviction is happening. Hence revert the
multithreaded cache flush support to simply the code and patch 3325002
adds gathering of pages support.
Bug 4628529
Change-Id: Ieb9237c6c2935cbf88036bc3ab83c33dc4d5e38d
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3325001
Tested-by: mobile promotions <svcmobile_promotions@nvidia.com>
Reviewed-by: Sachin Nikam <snikam@nvidia.com>
GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>
Reviewed-by: Pritesh Raithatha <praithatha@nvidia.com>
Reviewed-by: mobile promotions <svcmobile_promotions@nvidia.com>
Reviewed-by: svc-mobile-coverity <svc-mobile-coverity@nvidia.com>
On TOT, NvMap does page by page cache flush i.e. it takes virtual
address of each page present in the buffer and then perform cache
flush on it using dcache_by_line_op. This result in very poor
performance for larger buffers. ~70% of the time taken by
NvRmMemHandleAllocAttr is consumed in cache flush.
Address this perf issue using multithreaded cache flush
- Use a threshold value of 32768 pages which is derived from perf
experiments and as per discussion with cuda as per usecases.
- When the cache flush request of >= 32768 pages is made, then vmap
pages to map them in contiguous VA space and create n number of kernel
threads; where n indicate the number of online CPUs.
- Divide the above VA range among the threads and each thread would do
cache flush on the VA range assigned to it.
This logic in resulting into following % improvement for alloc tests.
-----------------------------------
Buffer Size in MB | % improvement |
----------------------------------|
128 | 52 |
256 | 56 |
512 | 57 |
1024 | 58 |
1536 | 57 |
2048 | 58 |
2560 | 57 |
3072 | 58 |
3584 | 58 |
4096 | 58 |
4608 | 58 |
5120 | 58 |
-----------------------------------
Bug 4628529
Change-Id: I803ef5245ff9283fdc3afc497a6b642c97e89c06
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3187871
Reviewed-by: Krishna Reddy <vdumpa@nvidia.com>
GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>
In Linux v6.13, commit b129125e1f96 ("arm64: asm-offsets: remove
DMA_{TO,FROM}_DEVICE") removes the definitions DMA_TO/FROM_DEVICE from
the assembly header file 'asm-offsets.h' which is used by the functions
__dma_map_area() and __dma_unmap_area() in nvmap_cache_maint.S.
From reviewing the NVMAP code, the function __dma_unmap_area() is never
used and so we can simply remove this. Split the __dma_map_area() into
two functions __dma_map_area_to_device() and
__dma_map_area_from_device() to avoid having to pass the direction.
Bug 4991895
Change-Id: I4a0f658401beff5c5e5457de72050b21acf820fa
Signed-off-by: Jon Hunter <jonathanh@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3261710
GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>
Reviewed-by: Ketan Patil <ketanp@nvidia.com>
When struct dma_coherent_mem* is typecasted to struct
dma_coherent_mem_replica* coverity complains with CERT EXP 39-C defect.
This defect says that we can not access a variable through a pointer of
an incompatible type.
Coverity is not able to detect that both the types essentially contain
the same members and also, we can not use struct dma_coherent_mem*
directly.
Add an union datatype to have both struct dma_coherent_mem* and struct
dma_coherent_mem_replica*. Union makes the coverity tool accept the
usage of both the types together.
Fix 4 CERT EXP39-C defects.
Bug 4479044
Change-Id: Ia29ede9b47b3e1e110450f365e125cc2b77d2ee7
Signed-off-by: N V S Abhishek <nabhishek@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3262832
GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>
Reviewed-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-by: Sachin Nikam <snikam@nvidia.com>
- Query heap functionality belongs to nvmap_alloc unit, as heap is
managed by it. Hence move the function to query the heap to nvmap_alloc
unit.
- Move nvmap_get_user_pages function to nvmap_alloc unit as it is
relevant for nvmap_alloc unit.
- Move nvmap_dma_alloc_attrs/free_attrs functions to nvmap_alloc unit
as they are more relevant for nvmap_alloc unit.
- Move dma_coherent_mem_replica, nvmap_carveout_node structs to
nvmap_alloc unit.
- Cleanup unused macros from nvmap_priv.h
JIRA TMM-5694
Change-Id: I8884831771443de7db0e95c3b2dfc43c03f7c48e
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3214196
GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>
- Move macro definitions from nvmap_priv.h to nvmap_alloc unit wherever
required.
- Cleanup unnecessary macros.
- Add function to cleanup the memory allocated for debugfs_info for
iovmm. This was missed in the previous patch where the allocation for
debugfs_info is moved to dynamic memory allocation.
- Move nvmap page pool related data structs from nvmap_priv to
nvmap_alloc unit.
JIRA TMM-5621
Change-Id: I3b668b2d6182da1bf0d2034c66834efc02d3179f
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3203118
Reviewed-by: mobile promotions <svcmobile_promotions@nvidia.com>
Tested-by: mobile promotions <svcmobile_promotions@nvidia.com>
- Move all data structures from nvmap_heap.h header file to
nvmap_alloc_int.h file as they are owned by nvmap_alloc unit.
- Provide getter and setter functions to get or set the members of these
data structures.
- Provide forward declaration of such data structures.
- Remove nvmap_heap.h header file as nvmap_heap is part of the
nvmap_alloc unit and nvmap_alloc unit exposes nvmap_alloc.h as header
file to other units.
JIRA TMM-5621
Change-Id: I2c4dd95a1a1011e4a7c1b425aa7521c6f13202da
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3201354
Reviewed-by: Pritesh Raithatha <praithatha@nvidia.com>
GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>
As part of the nvmap_refactoring, add nvmap_alloc.h file which include
declaration for functions which are exposed by nvmap_alloc unit to other
units. Also, add nvmap_alloc_int.h file which include declaration for
functions which are internal to nvmap_alloc unit that can be called by
files within nvmap_alloc unit.
JIRA TMM-5621
Change-Id: Ie30e5e8a4f87591eb9c49a0a349f837a22726fa5
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3198546
Reviewed-by: mobile promotions <svcmobile_promotions@nvidia.com>
Tested-by: mobile promotions <svcmobile_promotions@nvidia.com>
