nvmap uses pid of group_leader task to indicate a client process. During
create_client operation, whenever any client with the same group_leader
pid already exists in clients list of nvmap_device, then nvmap
increments the count field of nvmap_client struct. Otherwise, create a
new nvmap_client. Both of the operations i.e. checking the list for
client and incrementing the counter happen inside lock. On the other
hand, during nvmap_release, first the counter is decremented and checked
if it's zero or not. If it's zero then the lock is taken and client is
removed from client list of nvmap_device. As both the operations i.e.
decrementing the counter value and removing client from list (if the
counter becomes 0) are not happening inside a lock, it's resulting into
the following data race scenario.
1) nvmap_release on existing client process 1
- decrement client's counter
- counter value has become zero
- client is yet to be removed from the dev->clients list
- context switch happen to __nvmap_create_client as another
namespace/thread with same with same group_leader pid is created.
2) __nvmap_create_client
- as the client with same pid exists in dev->client list, it
increments counter value to 1, instead of creating a new client struct.
- context switch happen to nvmap_release from step 1
3) nvmap_release
- It calls destroy_client and remove the client from dev->client
list.
- Completes rest of the operations in destroy_client and returns.
- Context switch to remaining operations from step 2
4) nvmap_release
- Now, when the nvmap_release will be called for the thread/namespace
which was created in step 2, then list_del operation would fail as the
client struct was already removed from dev->client list.
Fix the above issue by doing both operations i.e. decrementing the
counter value and removing the client struct from dev->client list in a
single lock.
Bug 4829958
Change-Id: I87ebbcb45b18114d0ec75520443bee010f88d59e
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3209794
(cherry picked from commit cc74d1fe1b)
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3210845
Reviewed-by: Brad Griffis <bgriffis@nvidia.com>
Reviewed-by: Amulya Yarlagadda <ayarlagadda@nvidia.com>
Tested-by: Brad Griffis <bgriffis@nvidia.com>
GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>
Upstream commit cdd9a571b7d8 ("fs/proc: move page_mapcount() to
fs/proc/internal.h") made page_mapcount() an internal function for Linux
v6.11. This function is used by the NVMAP driver and so implement this
function in the NVMAP driver directly instead. Note that newer kernels
implement folio_entire_mapcount() for compound page counts where as
older kernels use compound_mapcount(). Use conftest to detect which of
these functions is present in the kernel.
Bug 4749580
Change-Id: I6504448727d6b3e9f2caa8581a66aad464ae2426
Signed-off-by: Jon Hunter <jonathanh@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3180097
(cherry picked from commit f327656ea3)
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/3183926
GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>
Reviewed-by: Ketan Patil <ketanp@nvidia.com>
Deferred dmabuf unmapping is being removed from kernel.
So, add similar support to cache sgt in NvMap.
During map_dma_buf() call, NvMap will create a mapping and an sgt
corresponding to it. It will also cache this sgt.
When unmap_dma_buf() is called for same sgt, NvMap will not unmap
the mappings. It will simply return from there.
Next time when the mapping request comes for same dmabuf, it will
look for existing sgt in cache and return it. This significantly
reduces mapping overhead for same buffer when it's mapped and unmapped
multiple times.
Free the sgt and unmap only when corresponding buffer is freed. When
all references from a buffer are removed, dmabuf_release() will be
called where sgt will be freed.
Bug 4064339
Change-Id: I7ed767ecaaac7aa44e6576e701b28537b84986ec
Signed-off-by: Ashish Mhetre <amhetre@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/2925224
Reviewed-by: svcacv <svcacv@nvidia.com>
Reviewed-by: Sachin Nikam <snikam@nvidia.com>
GVS: Gerrit_Virtual_Submit <buildbot_gerritrpt@nvidia.com>
Add support for Serial Id feature which will be used by Nsight for
buffer tracking purpose. This feature expects a unique serial id per
buffer even if it is shared across multiple client processes.
Add following code:
- Create a new global counter field for serial id in nvmap device.
Initialize it to 0 when nvmap device is initialized.
- Introduce a new field for serial_id in nvmap_handle struct.
- When nvmap_handle is created, assign it's serial_id field with global
counter's value, and increment global counter.
- During NvRmMemQueryHandleParameters return this serial_id associated
with the handle.
- Do not decrement counter for serial_id even after freeing the handle.
Bug 4138373
Change-Id: Ic1fe22b082eefb352986f8fa44d4c38d186a366f
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nv-oot/+/2918510
Tested-by: mobile promotions <svcmobile_promotions@nvidia.com>
Reviewed-by: mobile promotions <svcmobile_promotions@nvidia.com>
get_task_struct increment the ref count over task struct, and it will be
decremented as part of put_task_struct. task_struct won't be freed
unless it's refcount becomes 0. Hence the missing put_task_struct in
nvmap code was resulting into kmemleak. Fix it by add this missing call.
Also, mutex_unlock was missing in one of the return path, add it.
Bug 3901618
Change-Id: I630eac19e628a549179a8ddaad86ad4d2c9b9a53
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nvidia/+/2837383
Reviewed-by: svcacv <svcacv@nvidia.com>
Reviewed-by: Ashish Mhetre <amhetre@nvidia.com>
Reviewed-by: Sachin Nikam <snikam@nvidia.com>
GVS: Gerrit_Virtual_Submit <buildbot_gerritrpt@nvidia.com>
Some of the libraries may be linked into different namespaces, but they
belong to same process. So each namespace can open the /dev/nvmap node
and call nvmap ioctls, instead of reusing the already opened /dev/nvmap
FD. Because of this, memory handles created by one namespace can't be
used by other namespaces, even though they belong to same process.
Fix this issue by following:
- When /dev/nvmap node is opened, check if there is already any nvmap
client with same pid, if yes use that client; otherwise, create new
nvmap client.
- Track the number of namespaces via count field.
- When /dev/nvmap node is closed, destroy the nvmap client only when
count reaches to 0.
Bug 3689604
Change-Id: I4c91db36b88e78b7526dd63b006e562c8f66c7ae
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nvidia/+/2819915
Reviewed-by: svc-mobile-coverity <svc-mobile-coverity@nvidia.com>
Reviewed-by: svc_kernel_abi <svc_kernel_abi@nvidia.com>
Reviewed-by: Krishna Reddy <vdumpa@nvidia.com>
Reviewed-by: Ivan Raul Guadarrama <iguadarrama@nvidia.com>
GVS: Gerrit_Virtual_Submit <buildbot_gerritrpt@nvidia.com>
On TOT, in nvmap probe function, platform_set_drvdata is called after
misc register. So nvmap node is created and clients can open it even
before platform driver data is set. Because of which, the call to
dev_get_drvdata returns NULL and BUG_ON condition is hit. Move misc
register call towards end of the nvmap probe, so that /dev/nvmap is
exposed once all initilization is completed.
As misc register being moved to end of probe, the functions which are
called before misc register and which make use of dev_user field from
nvmap_device for calling dev_err/dev_info function will see the dev
field as NULL. Hence replace such calls with pr_err/pr_info.
Bug 3853486
Change-Id: Ifdd8443812a621aceada7739cf9b02fcf00568b4
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nvidia/+/2803672
Reviewed-by: svc-mobile-coverity <svc-mobile-coverity@nvidia.com>
Reviewed-by: svc-mobile-cert <svc-mobile-cert@nvidia.com>
Reviewed-by: svc_kernel_abi <svc_kernel_abi@nvidia.com>
Reviewed-by: Sachin Nikam <snikam@nvidia.com>
GVS: Gerrit_Virtual_Submit <buildbot_gerritrpt@nvidia.com>
On TOT, in case of carveouts, nvmap performs cache flush operation
during carveout creation, buffer allocation and buffer release. Due to
cache flush for entire carveout at creation time, nvmap takes ~430 ms
for probe. This is affecting boot KPIs.
Fix this by performing cache flush only at buffer allocation time,
instead of carveout creation and buffer release. This is reducing nvmap
probe time to ~0.69 ms.
Bug 3821631
Change-Id: I54da7dd179f8d30b8b038daf3eceafb355b2e789
Signed-off-by: Ketan Patil <ketanp@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/c/linux-nvidia/+/2802353
Reviewed-by: svc-mobile-coverity <svc-mobile-coverity@nvidia.com>
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Reviewed-by: Sachin Nikam <snikam@nvidia.com>
Reviewed-by: Ashish Mhetre <amhetre@nvidia.com>
GVS: Gerrit_Virtual_Submit <buildbot_gerritrpt@nvidia.com>
