Comments (4)
I'm not sure if there will be much interest in this feature.
Agreed @Aloqeely - CSV reading in pandas is already very complex and I think we would like to simplify it where possible. This feature seems too niche to me to have it supported directly in pandas.
If I have a large dataset with 100K rows, and I want to select 1000 rows at random, I first need to load the whole dataset into my RAM
How does one determine which 1000 rows to choose without knowing the total number of rows? Wouldn't that require reading the entire file anyways?
from pandas.
Thanks for the request but it seems like there's not much interest in this feature so closing
from pandas.
I think you can use skiprows
for this (I did not test it)
def selectFirstNRandom(size: int):
selected_count = 0
def skip(index: int):
if selected_count >= size:
return True
skip_row = random.random() > 0.5
if not skip_row:
selected_count += 1
return skip_row
return skip
df = pd.read_csv('your_file.txt', skiprows=selectFirstNRandom(100))
from pandas.
I'm not sure if there will be much interest in this feature.
@twoertwein's skiprows
solution probably works, but I think using chunksize
and iterator
is cleaner, it's less random though.
it = pd.read_csv("test.csv", chunksize=1000, iterator=True)
chunk1 = it.get_chunk() # returns DataFrame with first 1000 lines
chunk2 = it.get_chunk() # returns DataFrame with the next thousand lines
For more information on the chunksize
and iterator
parameters, you can see the IO Tools user guide
from pandas.
Related Issues (20)
- ENH: Add Float128 support for groupby. HOT 1
- PERF: Excessive memory consumption in pd.read_parquet HOT 3
- BUG: Resampling to `"B"` frequency with `closed="right"` and `label="right"` adds empty bins
- BUG: ArrowNotImplementedError: Unsupported cast from int64 to null using function cast_null HOT 1
- ENH: Adding `skipna:bool` to Rolling.sum HOT 2
- BUG: Replace fails after NaN in a Series of `string` HOT 2
- DEPR: future.no_silent_downcasting option HOT 1
- BUILD: Python Docker Build Issues HOT 4
- DOC: Incorrect Documentation: pd.Dataframe.dropna HOT 2
- BUG: Spurious `FutureWarning` when using `pd.read_json()` HOT 2
- BUG: read_json silently ignores the dtype when engine=pyarrow HOT 2
- ENH: support the Arrow PyCapsule Interface on pandas.Series (export) HOT 5
- BUG: HOT 3
- BUG: "Bad CRC-32 for file'docProps/core.xml" for Read large Excel file HOT 1
- BUG: Incorrect float32/float64 comparison result HOT 7
- BUG: import has error HOT 2
- BUG: subtracting datetime series from datetime dataframe, or datetime dataframe from datetime series, raises TypeError or UFuncTypeError HOT 2
- BUG: read_csv's usecols type hint isn't match with list of strings HOT 2
- BUG: OverflowError: value too large to convert to int when manipulating very large dataframes HOT 6
- DOC: Wrong bug number in what's new v3.0.0 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pandas.