In gk20a_pm_shutdown(), we currently call __pm_runtime_disable()
which prevents h/w access to new requests made after shutdown() call
Also, once gk20a_pm_shutdown() completes, platform code will
just rail gate the GPU
But it is possible that some other thread is already accessing
h/w while shutdown() was triggered and this can result in hang
Hence, wait until all currently executing jobs are finished
before returning from gk20a_pm_shutdown()
Also, we need to wait for GPU's usage count to become 1 since
platform code will increase the usage count and then call
shutdown(). Hence usage count of 1 indicates that GPU is idle
Bug 200099940
Change-Id: I1f2457829e2737c07302d13f355353a30c3b4e67
Signed-off-by: Deepak Nibade <dnibade@nvidia.com>
Reviewed-on: http://git-master/r/734920
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Tested-by: Bharat Nihalani <bnihalani@nvidia.com>
Currently we execute pm_runtime_get_sync() and then
gk20a_scale_notify_busy() without checking return value of
pm_runtime_get_sync()
In case of shutdown of GPU is already initiate, we get
a hard hang due to this as per below sequence :
- one thread invokes GPU shutdown and then forcibly rail
gates the GPU
- another thread (unaware of shutdown) calls gk20a_busy()
- since runtime PM is disabled in shutdown path,
pm_runtime_get_sync() fails
- but we still go on running gk20a_scale_notify_busy() which
tries to access some GPU registers and hangs
Fix this by jumping to failure path in case
pm_runtime_get_sync() fails
Bug 200099940
Change-Id: I022f2dfa9408f640fb44e6f4b10a437688779c0a
Signed-off-by: Deepak Nibade <dnibade@nvidia.com>
Reviewed-on: http://git-master/r/738509
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Tested-by: Terje Bergstrom <tbergstrom@nvidia.com>
Implement NVGPU_AS_IOCTL_GET_BUFFER_COMPBITS_INFO for requesting info
on compbits-mappable buffers; and NVGPU_AS_IOCTL_MAP_BUFFER_COMPBITS,
which enables mapping compbits to the GPU address space of said
buffers. This, subsequently, enables moving comptag swizzling from GPU
to CDEH/CDEV formats to userspace.
Compbits mapping is conservative and it may map more than what is
strictly needed. This is because two reasons: 1) mapping must be done
on small page alignment (4kB), and 2) GPU comptags are swizzled all
around the aggregate cache line, which means that the whole cache line
must be visible even if only some comptag lines are required from
it. Cache line size is not necessarily a multiple of the small page
size.
Bug 200077571
Change-Id: I5ae88fe6b616e5ea37d3bff0dff46c07e9c9267e
Signed-off-by: Sami Kiminki <skiminki@nvidia.com>
Reviewed-on: http://git-master/r/719710
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Tested-by: Terje Bergstrom <tbergstrom@nvidia.com>
Allow querying and setting default betacb size via debugfs. For global buffers
the value takes effect upon first boot of GPU, and has no effect after that.
Bug 1628352
Change-Id: Ib63f4299249c41eab1b36cc501b525cc54211195
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-on: http://git-master/r/733328
With CONFIG_PM_GENERIC_DOMAINS_OF enabled, device reboot
was getting hung while shutting-down gk20a. It was
happening because genpd_dev_pm_detach() was railgating
gk20a while other thread was still accessing it.
So, assigning NULL to dev->pm_domain->detach for gk20a,
so that genpd_dev_pm_detach() is not called during gk20a
shutdown, which will not railgate it.
This patch will be reverted once we have clean shutdown
for gk20a.
Bug 200070810
Bug 200099940
Change-Id: Ie2e89ea01a98a9d4f2f68a3ab07b6923ffa374f6
Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-on: http://git-master/r/735455
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Tested-by: Bharat Nihalani <bnihalani@nvidia.com>
We have below race condition during __gk20a_do_idle()
and force_reset case :
- before execution of __gk20a_do_idle(), a process drops the last
usage count of GPU, which triggers GPU railgate process
- but before GPU is really railgated (there is 500 mS delay),
some process calls __gk20a_do_idle()
- in __gk20a_do_idle(), we first take railgate_lock
- then we check if GPU is already railgated or not
- since it is not railgated yet (due to 500 mS delay), this
returns false
- then we call pm_runtime_get_noresume() which just increases the
usage counter
- in this particular case, this call just increases usage count to
1 from 0, but whereas GPU is already on its way to railgate
- while we check if GPU usage count drops to one, GPU gets railgated
- now if we have force_reset=true case, we will end up calling
pm_runtime_get_sync() which will take railgate_lock lock _again_
and try to unrailgate GPU
- this causes a deadlock on railgate_lock
To fix this, use below sequence :
- take railgate_lock
- check if GPU is already railgated
- release railgate_lock
- call pm_runtime_get_sync() which will keep GPU active even if
railgating is already triggered
- take railgate_lock again to prevent unrailgate in futher process
Also, add more descriptive comments to explain the flow
Bug 1624537
Change-Id: I0febc65d7bfac03ee738be200cf321322ffbe5a6
Signed-off-by: Deepak Nibade <dnibade@nvidia.com>
Reviewed-on: http://git-master/r/719625
(cherry picked from commit 480284eda16e2b50ee6368bad3d15574e098b231)
Reviewed-on: http://git-master/r/719620
Reviewed-by: Sachin Nikam <snikam@nvidia.com>
If the clock is null, calling the reset function will crash the
kernel. So, don't call the reset function.
Change-Id: I37ef25c8dca67bec8bf6654eb6e275b866bdae53
Signed-off-by: Alex Van Brunt <avanbrunt@nvidia.com>
Reviewed-on: http://git-master/r/742361
Reviewed-by: Automatic_Commit_Validation_User
GVS: Gerrit_Virtual_Submit
Implement a new buddy allocation scheme for the GPU's VA space.
The bitmap allocator was using too much memory and is not a scaleable
solution as the GPU's address space keeps getting bigger. The buddy
allocation scheme is much more memory efficient when the majority
of the address space is not allocated.
The buddy allocator is not constrained by the notion of a split
address space. The bitmap allocator could only manage either small
pages or large pages but not both at the same time. Thus the bottom
of the address space was for small pages, the top for large pages.
Although, that split is not removed quite yet, the new allocator
enables that to happen.
The buddy allocator is also very scalable. It manages the relatively
small comptag space to the enormous GPU VA space and everything in
between. This is important since the GPU has lots of different sized
spaces that need managing.
Currently there are certain limitations. For one the allocator does
not handle the fixed allocations from CUDA very well. It can do so
but with certain caveats. The PTE page size is always set to small.
This means the BA may place other small page allocations in the
buddies around the fixed allocation. It does this to avoid having
large and small page allocations in the same PDE.
Change-Id: I501cd15af03611536490137331d43761c402c7f9
Signed-off-by: Alex Waterman <alexw@nvidia.com>
Reviewed-on: http://git-master/r/740694
Reviewed-by: Automatic_Commit_Validation_User
GVS: Gerrit_Virtual_Submit
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Tested-by: Terje Bergstrom <tbergstrom@nvidia.com>
Gpu boosting with input events is causing more gpu power consumption
than required. To avoid this, touch boot for gpu is disabled by not
registering gpu device for cfboost frame work. Current rail gate
entry/exit latencies are fast enough to give smooth user experience.
Bug 200087243
Change-Id: I18673d9c95a44ce9bee87e860b4edb29212dc766
Signed-off-by: Seshendra Gadagottu <sgadagottu@nvidia.com>
Reviewed-on: http://git-master/r/721989
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Tested-by: Terje Bergstrom <tbergstrom@nvidia.com>
Fixed the following sparse warnings by making below APIs static:
- gk20a.c: warning: symbol 'gk20a_pm_restore_debug_setting' was not declared.
Should it be static?
- gr_gk20a.c: warning: symbol 'gr_gk20a_rop_l2_en_mask' was not declared.
Should it be static?
- gr_gm20b.c: warning: symbol 'gr_gm20b_rop_l2_en_mask' was not declared.
Should it be static?
Bug 200067946
Change-Id: I334893bb6614171bff835d270716a7dd262c9ba7
Signed-off-by: Amit Sharma (SW-TEGRA) <amisharma@nvidia.com>
Reviewed-on: http://git-master/r/718756
Reviewed-by: Deepak Nibade <dnibade@nvidia.com>
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-by: Sachin Nikam <snikam@nvidia.com>
New members are added in nvgpu_gpu_characterstics to export more
information required specially from CUDA tools.
Change-Id: I907f3bcbd272405a13f47ef6236bc2cff01c6c80
Signed-off-by: Sujeet Baranwal <sbaranwal@nvidia.com>
Reviewed-on: http://git-master/r/679202
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Tested-by: Terje Bergstrom <tbergstrom@nvidia.com>
The current CUDA drivers have been using the regops to
directly accessing the GPU registers from user space through
the dbg node. This is a security hole and needs to be avoided.
The patch alternatively implements the similar functionality
in the kernel and provide an ioctl for it.
Bug 200083334
Change-Id: Ic5ff5a215cbabe7a46837bc4e15efcceb0df0367
Signed-off-by: sujeet baranwal <sbaranwal@nvidia.com>
Reviewed-on: http://git-master/r/711758
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Tested-by: Terje Bergstrom <tbergstrom@nvidia.com>
First, defining a new structure to support gk20a
power domain. Then making necessary modifications
to add so as to add DT support for gpu power-domain.
bug 200070810
Change-Id: I29e1c24b181e14743d3969103abfd1882d171f07
Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-on: http://git-master/r/668973
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Add "force_reset" flag to __gk20a_do_idle()
For real world use cases like VPR resizing, we cannot wait
for railgate_delay (which is 500 mS). Hence use CAR reset
for this use case. (this is done via gk20a_do_idle() API
with force_reset = true)
Some of the test cases make use of sysfs "force_idle" and
they expect GPU to be into really railgated state and
not in CAR reset.
Hence when called from sysfs, set force_reset = false.
When global flag "force_reset_in_do_idle" is set, it will
override local flags and force CAR reset case.
This is desired in cases where railgating is not enabled
Also, set force_reset_in_do_idle = false for GM20B since
railgating has been enabled for GM20B
Bug 1592997
Change-Id: I6c5af2977c7211ef82551a86a7c1eb51b8ccee60
Signed-off-by: Deepak Nibade <dnibade@nvidia.com>
Reviewed-on: http://git-master/r/711615
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Tested-by: Terje Bergstrom <tbergstrom@nvidia.com>
Pass always the directory structure to mm functions instead of
pointers to members to it. Also split update_gmmu_ptes_locked()
into smaller functions, and turn the hard
coded MMU levels (PDE, PTE) into run-time parameters.
Change-Id: I315ef7aebbea1e61156705361f2e2a63b5fb7bf1
Signed-off-by: Deepak Nibade <dnibade@nvidia.com>
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-on: http://git-master/r/672485
Reviewed-by: Automatic_Commit_Validation_User
Compression page size varies depending on architecture. Make it
129kB on gk20a and gm20b.
Also export some common functions from gm20b.
Bug 1592495
Change-Id: Ifb1c5b15d25fa961dab097021080055fc385fecd
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-on: http://git-master/r/673790
Add error prints in gk20a_do_idle() to narrow down
the failure point
Bug 200064302
Change-Id: Iffe1151bdc200a79b88e273b3b01523f8e46d130
Signed-off-by: Deepak Nibade <dnibade@nvidia.com>
Reviewed-on: http://git-master/r/664446
(cherry picked from commit bf1cd9b5551d27cb5cc468795cd147376f48e482)
Reviewed-on: http://git-master/r/666218
Reviewed-by: Sachin Nikam <snikam@nvidia.com>
Remove support for gk20a sparse textures. We're using implementation
from user space, so gk20a code is never invoked.
Also removes ref_cnt for PTEs, so we never free PTEs when unmapping
pages, but only at VM delete time.
Change-Id: I04d7d43d9bff23ee46fd0570ad189faece35dd14
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-on: http://git-master/r/663294
To run CUDA apps, the following minimal changes have been
made:
- power-gating is disabled for vgpu
- regop rd/wr returns -ENOSYS
Tools (debugger/profiler) support is known to not work and
not needed at this time.
Bug 200043227
Change-Id: I923caad78450e72d310fb9290cf2849ed5460ad5
Signed-off-by: Aingara Paramakuru <aparamakuru@nvidia.com>
Reviewed-on: http://git-master/r/592878
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
When removing the module, remove the device from devfreq and free
resources allocated when scaling is initialized.
Bug 1476801
Change-Id: I7bb0f8112a5bf7e5ce2fc56cf8af7059d910002c
Signed-off-by: Konsta Holtta <kholtta@nvidia.com>
Reviewed-on: http://git-master/r/594444
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Add ->remove() for undoing the ->probe() and ->late_probe() in
gk20a_platform devices, and call it when gk20a is removed.
Bug 1476801
Change-Id: Ic9b29c0a7ea4a4cae7b5a0f66774bd799eb28434
Signed-off-by: Konsta Holtta <kholtta@nvidia.com>
Reviewed-on: http://git-master/r/594443
Reviewed-by: Automatic_Commit_Validation_User
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Fix gk20a module removal so that the driver can be correctly removed
when built as a module (CONFIG_GK20A=m). Reinsertion correctly will need
more missing deinit calls, which are not yet implemented.
This change has no effect yet on any builds since the module is built in
by the defconfig.
- don't free threaded irqs that are bound to the device
- destroy cde if it is enabled
- remove all debugfs entries recursively generated directly and by cde,
pmu, clk etc.
- free secure buffer if it exists
- remove pm_runtime_put, since it doesn't have a pairing get
- pm_runtime defines proper function stubs if it is not enabled, so
remove ifdefs and query pm_runtime_enabled()
- null and free gk20a only after all deinit has been done
Bug 1476801
Change-Id: I73ad72832bdb501fd7071d6ac68d461ee63a760d
Signed-off-by: Konsta Holtta <kholtta@nvidia.com>
Reviewed-on: http://git-master/r/594442
Reviewed-by: Automatic_Commit_Validation_User
GVS: Gerrit_Virtual_Submit
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Query interrupt number and reset id from HW. Use the number
from HW when enabling and detecting interrupts.
Bug 200036089
Bug 1567274
Change-Id: If9cb4db79a19dcb193ba7ad9db7081f4fe1ab433
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-on: http://git-master/r/600988
Add the following info into GPU characteristics: available big page
sizes, support indicators for sync fence fds and cycle stats, gpc
mask, SM version, SM SPA version and warp count, and IOCTL interface
levels. Also, add new IOCTL to fetch TPC masks.
Bug 1551769
Bug 1558186
Change-Id: I8a47d882645f29c7bf0c8f74334ebf47240e41de
Signed-off-by: Sami Kiminki <skiminki@nvidia.com>
Reviewed-on: http://git-master/r/562904
Reviewed-by: Automatic_Commit_Validation_User
GVS: Gerrit_Virtual_Submit
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
In gk20a_do_idle(), we wait for platform->railgate_delay
to allow GPU to go into rail gate
But sometimes we set platform->railgate_delay = INT_MAX
to disable GPU rail gating but allow it to suspend during
low power state
Due to this, force_idle API fails (it waits for INT_MAX)
To fix this, allow forcing CAR reset instead of rail gating
with flag "force_reset_in_do_idle" defined in gk20a_platform
Set this flag for gm20b until we fix the railgate_delay
Bug 1517584
Change-Id: I031aa56f87d4db3727e2c3a3e5eeaf18503dd449
Signed-off-by: Deepak Nibade <dnibade@nvidia.com>
Reviewed-on: http://git-master/r/593704
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Tested-by: Terje Bergstrom <tbergstrom@nvidia.com>
Do not call PMU ELPG calls if ELPG should be disabled. Also skips
initialization of PMU ucode if PMU is disabled.
Bug 1567274
Change-Id: Ia9cd3b553c358142ee05a1b0e0832f9412f7cf17
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-on: http://git-master/r/593335
Reviewed-by: Automatic_Commit_Validation_User
Reviewed-by: Seshendra Gadagottu <sgadagottu@nvidia.com>
Fix below sparse warnings :
warning: Using plain integer as NULL pointer
warning: symbol <variable/funcion> was not declared. Should it be static?
warning: Initializer entry defined twice
Also, remove dead functions
Bug 1573254
Change-Id: I29d71ecc01c841233cf6b26c9088ca8874773469
Signed-off-by: Deepak Nibade <dnibade@nvidia.com>
Reviewed-on: http://git-master/r/593363
Reviewed-by: Amit Sharma (SW-TEGRA) <amisharma@nvidia.com>
Reviewed-by: Automatic_Commit_Validation_User
Reviewed-by: Sachin Nikam <snikam@nvidia.com>
During gpu suspend, cancel all pending delayed cde work
to avoid issues of scheduling this delayed work
during suspend/resume when gpu is not ready.
Bug 1574000
Change-Id: I2b6bfa489435a781dc576a077f9af01b1e1628ce
Signed-off-by: Seshendra Gadagottu <sgadagottu@nvidia.com>
Reviewed-on: http://git-master/r/593557
Reviewed-by: Shridhar Rasal <srasal@nvidia.com>
Reviewed-by: Deepak Nibade <dnibade@nvidia.com>
Reviewed-by: Prashant Gaikwad <pgaikwad@nvidia.com>
Tested-by: Prashant Gaikwad <pgaikwad@nvidia.com>
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
To assert reset on GPU, we store "gpu_ref" clock in
platform->clk[0] and use it to assert/deassert reset
But for gm20b, "gpu_ref" is no longer resettable.
To fix this, add two callbacks in gk20a_platform :
.reset_assert and .reset_deassert
Also, add a pointer "clk_reset" to store the clock
which needs to be reset
For gk20a specific implementation, we continue to
reset platform->clk[0]
For gm20b specific implementation, we first request
"gpu_gate" clock, store it and use it to assert reset
Bug 1513685
Bug 1517584
Change-Id: I15a583a4a07eb663b442084be8b8c7d0c7c7a142
Signed-off-by: Deepak Nibade <dnibade@nvidia.com>
Move accesses to MC registers under HAL so that they can be
reimplemented per chip.
Do chip detection and HAL initialization only once.
Bug 1567274
Change-Id: I20bf2f439d267d284bfd536f1a1dfb5d5a2dce4c
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-on: http://git-master/r/590385
CONFIG_GK20A_CDE has not even been used for enabling CDE, just for
initializing it at boot time, and it was disabled; initialization has
been done late when the engine is first used. Remove the config
setting and add information about CDE support in gk20a platform data,
forcing the initialization at boot time. Boot time init removes rare
race conditions when CDE would be initialized by first user.
Bug 200046882
Change-Id: I85d5fb73dc27acbbe203138d25f6e342de030d93
Signed-off-by: Konsta Holtta <kholtta@nvidia.com>
Reviewed-on: http://git-master/r/562855
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Tested-by: Terje Bergstrom <tbergstrom@nvidia.com>
Convert GR and LTC HALs to use const structs, and initialize them
with macros.
Bug 1567274
Change-Id: Ia3f24a5eccb27578d9cba69755f636818d11275c
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-on: http://git-master/r/590371
Enable clocks only if they are defined. This prevents panic in cases
where clock does not need to be enabled explicitly.
Bug 1567274
Change-Id: I7113c6d874b61acc2646effda9c02d3d1817c531
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Do not build clock code if TEGRA_CLK_FRAMEWORK is not defined. Also
make GK20A_DEVFREQ depend on TEGRA_CLK_FRAMEWORK, and build scaling
governor only if GK20A_DEVFREQ is enabled.
Bug 1567274
Change-Id: I6ea1462e7a110fb46c9d66ceda71167cff19699e
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-on: http://git-master/r/562475
gk20a_get_client() and gk20a_put_client() routines are effectively dead
code. The GPU has been using pm_runtime for reference counting whether
the device should be turned on or off, and gk20a_get_client() and
gk20a_put_client() have had no positive effect on the behaviour.
In worst case these functions trigger some issues as they may trigger
code paths that should not be run. There is also a race between get/put
and busy/idle.
This patch removes the functions and reworks as_gk20a.c to correctly use
gk20a_busy()/gk20a_idle() where put/get was required.
Additionally, finalize_poweron() is moved to gk20a_busy(), similarly as
it was with gk20a_get_client(). If pm_runtime is not in use, the device
is only powered on and never off. Currently this affects vgpu power
management since it does not use pm_runtime yet.
Bug 1562096
Change-Id: I3162655f83457e9caccd9264eed36b5d51e60c52
Signed-off-by: Arto Merilainen <amerilainen@nvidia.com>
Signed-off-by: Konsta Holtta <kholtta@nvidia.com>
Reviewed-on: http://git-master/r/414998
Reviewed-by: Automatic_Commit_Validation_User
Reviewed-by: Seshendra Gadagottu <sgadagottu@nvidia.com>
Tested-by: Seshendra Gadagottu <sgadagottu@nvidia.com>
GVS: Gerrit_Virtual_Submit
Reviewed-by: Terje Bergstrom <tbergstrom@nvidia.com>
Interrupt mask init state is correct, so do not touch it.
Bug 1567274
Change-Id: I70673e406944823bd1cbfeee93ec75ce1e1af5da
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
Reviewed-on: http://git-master/r/558292
Merge initialization code from gk20a_init_system_vm(),
gk20a_init_bar1_vm() and gk20a_vm_alloc_share() into gk20a_init_vm().
Remove redundant page size data, and move the page size fields to be
VM specific.
Bug 1558739
Bug 1560370
Change-Id: I4557d9e04d65ccb48fe1f2b116dd1bfa74cae98e
Signed-off-by: Terje Bergstrom <tbergstrom@nvidia.com>
