Is there a downside to holding onto the same pools for the lifetime of the app? If we Return memory, but never Clear the pool, will it leak?
A leak could happen if you let a BufferPool go out of use while it retains pinned references to memory. If you intend to let a BufferPool get garbage collected, Clear it first.
If you keep using a BufferPool for the lifetime of the application and let the OS process teardown do its thing without calling Clear, there will be no leak because process teardown doesn't much care about what the GC thinks. You might still get an assert from the BufferPool if the debug finalizer runs before teardown, but it's ignorable in that case since the OS is gobbling everything anyway.
Should we construct the pool with smaller minimumBlockAllocationSize and expectedPooledResourceCount if it will only be used for 1 or a few objects?
Maybe- there's a bit of difficulty saying 'yes' because smaller allocations won't be put into the large object heap, but will still be pinned. That could interfere with the GC more than a LOH allocation that the GC probably won't move anyway.
If the lifespan of the pool (and its pins) is very short, the GC wouldn't be too bothered either way. Constantly recreating buffer pools with large backing allocations would probably hit some zero initialization costs, I suppose.
Clearing would clean up the memory immediately and we would only want to call this when we're done using the resource. What does setting Pinned to false do, and when would we do that -- when loading the resource or after we're done using it?
Setting Pinned to false allows the GC to move the memory referenced by the buffer pool, if it wants to. This means any outstanding buffers will become invalid, because the memory they pointed to might have moved.
In the rare case where 1) there are no outstanding buffers and 2) you want to perform a GC and 3) you want the memory backing the buffer pool to be potentially compacted (perhaps because you're forcing a LOH compaction) and 4) you want to keep the previously allocated internal pools around, then setting Pinned to false for a little while makes sense. You can't take resources from the pool until it's Pinned again.
Unless you need to do exactly that, I would generally advise ignoring the existence of the Pinned property. It'll likely go away once the library moves off of .NET Standard 2.0 (in the .NET 5-or-later timespan).
On to general recommendations:
I'd say just do what's easiest and simplest. Avoid anything that involves locking multithreaded access, since that is effectively never the easiest. Having a pool associated with a worker thread that it uses for ephemeral allocations, regardless of what the worker thread is working on, is one easy way to handle things.
If possible and convenient, avoid the complexity of tracking which buffer pool a given allocation came from. If you're creating a bunch of Mesh shapes from different pools, you'll need to know which buffer pool allocated what when they get disposed. You could store metadata for that, but it's a bit of a pain.
In many cases, persistent allocations can be preallocated by one central pool without locking, and then worker threads can use those allocations. This is often the simplest approach when it's possible. Ephemeral allocations could still be done on per-thread pools without adding complexity since the allocation will get returned to the thread pool before the worker finishes.