Comments (21)
In core.matrix / generic array functionality the expected behaviour would be:
- first on a row matrix [[1 2 3]] returns a vector [1 2 3]
- first on a column matrix [[1] [2] [3]] returns a length one vector [1]
That is, first is always consistent with taking the first slice of the first (row) dimension.
from clatrix.
Yes - I also thinking about this approach... I've tried to workaround this in to-list
function, but that also requires changes on the Incanter's side.
from clatrix.
I'm mildly trying to discourage first
/ map
/ seq
etc as applied to matrices arrays BTW: they are convenient sometimes but won't work if/when we introduce new matrix types from the Java world that don't exactly fit Clojure's conventions. In particular, if a class implements Iterable
in some way then that's the behaviour you will get, like it or not.
Better, I think, to move to the protocol-backed equivalents in core.matrix
so we can guarantee behaviour that is both consistent and extensible.
from clatrix.
we need to discuss this problem more deeply, as it will heavily affect Incanter's functions.
from clatrix.
I know - it's a tricky one! The good news is that it doesn't have to be a big bang switchover, we can support both in parallel I think. Anyway, better to continue discussion on the relevant mailing lists.
from clatrix.
I also have a related problem:
boltzmann.jblas> (seq (matrix [[1 2] [3 4]]))
( A 1x2 matrix
1.00e+00 2.00e+00
A 1x2 matrix
3.00e+00 4.00e+00
)
boltzmann.jblas> (seq (matrix [[1 2]]))
(1.0 2.0) ; seq of doubles
To make the single row case compatible with the other core.matrix/jblas routines, I had to convert the rows back to the matrix type explicitly (because ISliceWrapper of rows isn't countable and again not exchangable with the matrix type), which is somewhat ugly and adds probably some overhead (since all data has to pass through this copying code for each training epoch):
(map (comp mat/matrix vector)
(rows v-probs-batch))
source
I can optimize that probably, but it also took some time to figure it out and I had such problems also with the special core.matrix Vector type some times in the past, since most jblas routines expect matrices and the Vector type is not really compatible.
I understand the Iterable interface problem which probably b0rks the jblas types, so maybe if the ISliceWrapper object would behave like a Matrix I would be fine. I am not sure whether core.matrix.Vector is a good idea with jblas.
from clatrix.
@ghubber I assume you mean the clatrix.core.Vector
type? core.matrix doesn't have a Vector type specifically.
The clatrix.core.Vector type actually uses a JBlas matrix under the hood, so it should work fine for all the JBlas interop. Having said that, I think a lot of this code hasn't quite received all the testing it needs for all the interoperability cases. If anyone fancies doing some test.check
generative testing that would be cool :-)
from clatrix.
Ok, stupid question. Why do we need this Vector type then? The problem, if I recall correctly, was that the operations like Matrix-multiplication return 1-column/row Matrices, so when I loop the type changes.
from clatrix.
Conceptually 1-D vectors are different from 2D matrices even if they have the same elements.
It's the same as the difference between [[1 2 3]]
and [1 2 3]
.
Clatrix doesn't strictly need its own vector type. It could use Clojure vectors or something else in cases where it needs a 1D vector. The main requirement is that it is able to produce and consume 1D arrays where needed by the core.matrix API.
from clatrix.
You are right ofc., I got confused by the introduction of a separate Vector type. I am just working on a core.matrix
implementation for nd4j
, because I want to have fast GPU based deep learning in Clojure. I am not sure yet how consistent the deeplearning4j code is (I know @mikera had some discussions about ndarrays with them), but I don't see the Clojure community reinvent all the work there atm.
In core.matrix / generic array functionality the expected behaviour would be:
first on a row matrix [[1 2 3]] returns a vector [1 2 3]
first on a column matrix [[1] [2] [3]] returns a length one vector [1]That is, first is always consistent with taking the first slice of the first (row) dimension.
Yes, that would be consistent iteration over ndarrays. A peculiar thing is to expect unboxed numbers when iterating over vectors, since this turns an Iterator<NDArrayThing>
into Iterator<Object>
in Java terms. But I really don't understand why the clatrix classes return a scalar on (first (clatrix/matrix [[1 2]]))
, since Clatrix wraps JBlas' DoubleMatrix.
Also couldn't you get the Iterable added in upstream JBlas to avoid wrapping? (for opensource libs this is at least possible). This makes seamless interaction with APIs written for JBlas possible and I try to get this for nd4j (maybe it is stupid?).
https://github.com/deeplearning4j/nd4j/pull/374
DoubleMatrix
does not implement Iterable yet and can also store vectors. Why do you need a separate vector type? In general ISeq should be a protocol like in cljs, then we wouldn't have all that trouble, but this is a whole new story...
(I think an Iterator over elements for a tensor with dimensions is really weird and should be part of the API, not exposed by Iterable for any Java vector lib. Just my 2 pence.)
from clatrix.
@whilo an nd4j core.matrix implementation would be great!
However did you take a look at Vectorz / vectorz-clj? I think that implementing GPU support for Vectorz (via netlib-java) would actually be a pretty good way to get GPU matrix support in Clojure. And Vectorz has some advantages:
- It's the most mature core.matrix implementation
- It is very fast on the pure Java side for 1D vector maths, which don't really benefit much from GPUs but are very central to deep learning techniques
- It has a lot of specialised array implementations (diagonal matrices, sparse vectors, specialised sub-array types etc.) that give big performance enhancements for many applications when used appropriately.
I actually started an implementation here and it seems to work:
https://github.com/mikera/vectorz-native
Anyway I'd be interested to compare performance of the nd4j based implementation with vectorz-clj, something along the lines of http://core.matrix.bench-t2.large.s3-website-eu-west-1.amazonaws.com/554c2ae93f357522cca7e383e7ad90fef451c139.html
from clatrix.
@whilo have you got a public repo yet for your nd4j core.matrix implementation? I'm interested to take a look and try it out!
from clatrix.
It was only a weekend hack, because I would like to have competitive Clojure implementation to theano: https://github.com/whilo/clj-nd4j
I stopped working on it once I figured out I had to wrap the ndarray class. But since this is out of the way now, I would try to get the compliance tests passing.
from clatrix.
For deeplearning the absolute bottle-neck in my experience is matrix multiplication between batches of training samples and weight matrices. What do you mean by
1D vector maths, which don't really benefit much from GPUs but are very central to deep learning techniques
?
from clatrix.
I've done quite a bit of deep leaning using only 1D vectors and sparse operations (which don't require full matrix x matrix multiplication). Agree that matrix multiplication will be the bottleneck if you are using big dense matrices, but that isn't always required.
from clatrix.
@whilo thanks for sharing! I'll take a look and see if I can get the compliance tests passing. You shouldn't need to wrap the ND4J INDArray, having taken a quick look I think it has all the functionality required to work as an N-dimensional array.
from clatrix.
Good, thank you! I will have a look into the issue you pointed out in the pull-request. Most models I have seen so far have dense matrices. Which models have you trained? Backprop then only needs to update non-zero weights so matrices stay sparse, right?
from clatrix.
Correct. It is a very efficient way to have a lot of feature detectors, but without exploding the number of weights.
from clatrix.
Interesting. I have used dropout a bit, which is really nice regularization. Do you set the sparsity at the beginning of training or is it adaptive by pruning small weights and speculatively "forming synapses". (I work on biologically inspired neuron models and port deep learning techniques for my master, Boltzmann machines so far, to them.) In biology synapses form all the time.
from clatrix.
I've played a bit with both..... seems that a fixed sparsity works fine since the nodes just specialise towards whatever they have available as input. The pruning / forming new synapses also seems to work and performs better in some cases but not sure it it is always worth the extra complexity and overhead.
Of course, if you are doing stuff like convolutional networks it really helps to have a sensible hypothesis about how nodes should be connected in advance.
from clatrix.
Back to the original issue, do you think this is fixable in Clatrix? We probably should not discuss this here :).
from clatrix.
Related Issues (20)
- 0987 HOT 1
- Make as-vec work consistently on all matrix sizes
- Make as-vec work consistently on all matrix sizes
- NPE when comparing 2 empty matrices
- rank calculated incorrectly
- Would you consider a pr for get and submatrix? (:Question) HOT 4
- Can't reshape matrix
- Element map (emap) on matrix returns LazySeq HOT 4
- Race condition in loading implementation
- reshape still not working clatrix 0.3.0? HOT 8
- Inconsistency in handling seq/map/filter, etc.
- Jblas SVD incorrect result HOT 1
- 0 and 1 argument results not working for + HOT 1
- Documentation HOT 2
- Clatrix not transposing V* matrix in correctly in SVD return HOT 9
- thought about switching to neanderthal HOT 1
- Missing support for PMatrixProducts HOT 1
- svd return value doesn't match Vectorz's svd
- warning related to `abs` function when using Clojure 1.11 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clatrix.