Comments (4)
JulienPeloton
commented on June 19, 2024
Problem solved in #55 ! For the record, after the fix, the DAG is:
No more explicit union! Under the hood it uses PartitionedFile but the only things I had to do was to tell Spark directly that I have many files... (and not creating RDD one-by-one + union...).
from spark-fits.
JulienPeloton
commented on June 19, 2024
Still need to understand the error when reading >> 10000 files at once... Keeping this issue open.
from spark-fits.
ptallada
commented on June 19, 2024
Hi!
Is there any fix yet for reading > 10000 files yet?
Thanks!
from spark-fits.
JulienPeloton
commented on June 19, 2024
Hi @ptallada,
Unfortunately there has been no work on this recently - and I have little time these days.
Eventually, this would be solved when spark-fits moves to Spark Datasource V2 (stalled work for the moment).
from spark-fits.
Related Issues (20)
- Slashes in TTYPEn are not handled HOT 5
- Switch the CI to GitHub actions
- header challenge: cannot infer size of type B from the header HOT 5
- FITS HDU image: no bijection between TFORMn value (L, A, B, J, K, ..) and `BITPIX`
- DataFrame reader cannot load files as list of paths or comma separated string of paths HOT 5
- Read data from s3 HOT 2
- Benchmarking on s3
- header challenge: header with multivalued columns raises "java.lang.ArithmeticException: / by zero" during df.select(any_column).show() or df.select(any_column).take() HOT 5
- Long row size (> 1KB): java.lang.ArithmeticException: / by zero HOT 1
- Add support for Scala 2.12 HOT 1
- .option('columns', 'col1,col2,col3,col4') does not preserve order HOT 3
- Bad mapping between scalars (SPARK) and 1element vector (FITS) HOT 1
- fits files that match glob expressions which might contain empty HDU tables (NAXIS=0) causes fatal error HOT 4
- subset of rows from multiple DoubleType columns are incorrectly loaded as very small/large values HOT 12
- some columns are missing in dataframe when loaded with spark-fits (although present when loaded with topcat/astropy/fitsio) HOT 1
- CI: Install of Oracle JDK8 is failing HOT 1
- Relationship between SparkFits dataframe and source fits file? HOT 13
- spark-fits write mode HOT 2
- spark-fits produces duplicated rows when reading some fits files HOT 7
- Nans are present even after filtering out nans when reading large fits files (~1GB) HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-fits.