Comments (3)
from arkdb.
Hello.
Yes, one test does use future.apply to check parallel reads, but doesn’t set the number of cores.
It could be that testthat somehow circumvents R CMD check
detection of parallelly::availableCores()
; I created HenrikBengtsson/parallelly#109 to investigate if this could be the case.
Some tests may be using duckdb now, which is threaded by default.
Ah, that could be it. From the htop
screenshot, the child processes looks like forked processes (e.g. mclapply, "multicore"), but it could be that threads appear like that too. I'm quite sure it's not background R workers (e.g. PSOCK, "multisession"), because they'd have another set of command-line options.
Do you have a writeup on your rant?
Definitely; https://www.jottr.org/2022/12/05/avoid-detectcores/
You are familiar with ‘nice’ or containers / linux jails to control cpu use? Usually I find ram is the issue on shared
machines…
Yes; I can technically, protect others against this, and eventually our system will get cgroups2 protection others against this too. That would only protect others from overuse, but I would still overuse, resulting in lost of context switching, and more so the more CPU cores there are. And, in the bigger picture, this won't help anyone else out there. My blog post gives most of my arguments.
from arkdb.
thanks, I agree with all your points that are specific to detectCores()
-- as you see we're not setting that. (There are also reasons when a user might want to set this higher than the number of actual cores, e..g. when I/O is limiting). I'm less clear how your take applies to other tools, specifically lower-level libraries that are threaded by default (duckdb, arrow, gdal, openblas, loads of ml stuff etc). Most of these things are not impacted by the number of connections. I agree that connections in R can be a limited resource, but I don't quite understand the case that cpu use itself is limiting... anyway we are probably on the same page, I'm just a bit unclear how you feel about the default-threaded C-level applications like duckdb. To me, I think these are reasonable defaults that cause no real risk.
anyway, thanks for opening an issue and for all you do for the R community!
from arkdb.
Related Issues (20)
- Timeline for CRAN release HOT 1
- arkdb requires R (>= 3.5.0) HOT 2
- unark should permit custom table names and/or enforce SQL-valid names
- Support connections from pool HOT 1
- support parallelization? HOT 1
- Implement native bulk importers where possible HOT 1
- Set readr method's default column type to `character`? HOT 2
- Pass `na` argument to readers HOT 2
- Handling column types robustly HOT 16
- Injecting filters HOT 1
- Default arguements for streamable_parquet HOT 1
- duckdb filename adjustments HOT 1
- Need short vignette for new functionality
- Adding a callback function for data adjustment HOT 4
- installation error HOT 2
- `error in evaluating the argument 'name' in selecting a method for function 'dbExistsTable'` HOT 1
- Different number of rows between the original table and the new imported one. HOT 2
- Importing the local .tsv.bz2 file via unark() to a remote AWS RDS MySQL Database is taking an unusual amount of time HOT 9
- Possible issue with OFFSET / LIMIT query HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arkdb.