juliaai / mlflowclient.jl Goto Github PK
View Code? Open in Web Editor NEWJulia client for MLFlow.
Home Page: https://juliaai.github.io/MLFlowClient.jl/
License: MIT License
Julia client for MLFlow.
Home Page: https://juliaai.github.io/MLFlowClient.jl/
License: MIT License
https://github.com/rejuvyesh/MLFlowLogger.jl might be of interest.
I am creating a new run with the code below
mlf = MLFlow("http://localhost:5000")
mlf_experiment = getorcreateexperiment(mlf, "experiment_name")
mlf_run = createrun(
mlf,
mlf_experiment,
tags=[
Dict(
"key" => "mlflow.runName",
"value" => "run_name"
)
]
)
which fails with the following error message:
ERROR: LoadError: MethodError: no method matching parse(::Type{Int64}, ::Int64)
Closest candidates are:
parse(::Type{T}, ::AbstractChar; base) where T<:Integer at parse.jl:40
parse(::Type{T}, ::AbstractString; base) where T<:Integer at parse.jl:240
parse(::Type{T}, ::AbstractString; kwargs...) where T<:Real at parse.jl:379
Stacktrace:
[1] MLFlowRunInfo(info::Dict{String, Any})
@ MLFlowClient ~/.julia/packages/MLFlowClient/Yj8Aj/src/types.jl:124
[2] MLFlowRun
@ ~/.julia/packages/MLFlowClient/Yj8Aj/src/types.jl:224 [inlined]
[3] createrun(mlf::MLFlow, experiment_id::Int64; start_time::Missing, tags::Vector{Dict{String, String}})
@ MLFlowClient ~/.julia/packages/MLFlowClient/Yj8Aj/src/runs.jl:23
[4] createrun(mlf::MLFlow, experiment::MLFlowExperiment; start_time::Missing, tags::Vector{Dict{String, String}})
@ MLFlowClient ~/.julia/packages/MLFlowClient/Yj8Aj/src/runs.jl:30
I have fixed this by adding the following code before importing MLFlowClient
import Base.parse
function Base.parse(::Type{Int64}, val::Int64)
return val
end
using MLFlowClient
I have traced the error to the constructor call of MLFlowRunInfo (https://github.com/JuliaAI/MLFlowClient.jl/blob/main/src/types.jl#L115) which receives a dictionary.
In my case, the HTTP response dictionary contains a start_time entry of type a Int64, causing the subsequent parse call to fail.
This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml
to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix
on this issue.
I'll open a PR within a few hours, please be patient!
The current code requires that tags must be defined as:
[Dict("key" => "foo", "value" => "bar"), Dict("key" => "missy", "value" => "gala")]
but what about making all the logic behind to allow the user define the tags as:
# as an array of pairs
["foo" => "bar", "missy" => "gala"]
# as a key-value singular dict
Dict("foo" => "bar", "missy" => "gala")
Since MLFlow only supports string parameters, these must be automatically converted before sending to the POST endpoint.
This allows us to do:
logparam(mlf, run, :key, 5)
instead of
logparam(mlf, run, "key", "5")
According with this part
dagshub mlflow integrations
how to fill params of headers
mlf = MLFlow("https://dagshub.com/math4mad/mlj-test.mlflow";headers=Dict("MLFLOW_TRACKING_USERNAME"=>"******","MLFLOW_TRACKING_PASSWORD"=>"********" ))
# Initiate new experiment
experiment_id = createexperiment(mlf; name="price-paths")
# Create a run in the new experiment
exprun = createrun(mlf, experiment_id)
follow Set-up your credentials setting name and token , now it not working
The context of this proposal is this synchronisation issue.
The main problem with logging in parallelized operations is simply this: requests are
posted directly to an MLflow service without full information about the state the service
at the time the request is ultimately acted on. I propose we resolve this as follows:
Instead of a client posting requests directly to an MLflow service, they are posted
(put!
) to a first-in-first-out queue (Julia Channel
). Requesting calls will return
immediately, unless the queue is full. In this way, the performance of the parallel
workload is not impacted.
A single Julia Task
dispatches requests (take!
s) from the end of the queue. Whenever
a request has the possibility of altering the service state (e.g., creating an
experiment), then the dispatcher waits for confirmation that the state change is
complete before dispatching the next request.
I imagine that we can insert the queue (buffer) without breaking the user-facing
interface of MLFlowClient.jl.
I have implemented a POC for this proposal and shared it with two maintainers, and can share with anyone else interested.
Testing this package in the usual way, on one's local machine, does not work:
using Pkg
Pkg.activate(temp=true)
Pkg.add("MLFlowClient")
Pkg.test("MLFlowClient")
Testing Running tests...
createexperiment: Error During Test at /Users/anthony/.julia/packages/MLFlowClient/Szkbv/test/test_experiments.jl:1
Got exception outside of a @test
HTTP.Exceptions.StatusError(403, "POST", "/api/2.0/mlflow/experiments/create", HTTP.Messages.Response:
"""
HTTP/1.1 403 Forbidden
Content-Length: 0
Server: AirTunes/620.8.2
Obviously the error thrown in not helpful. I expect one needs an active MLflow service running on your machine, and that it must have the appropriate uri.
Here is a related issue with a suggestion for remedy: JuliaAI/MLJFlow.jl#20
Hi,
in the examples I see that it's possible to pass some dictionary with Authentication token, does that mean authentication is supported? if I dont have basic-auth set, the instance gets hacked. So it would be good to have.
As per https://github.com/JuliaAI/MLFlowClient.jl/blob/a31b41a91dc32107fa7dcd6ad5a581c116a4ff9c/src/loggers.jl#L69-72 we currently can't upload artifacts to remote servers
Assumes that artifact_uri is mapped to a local directory.
At the moment, this only works if both MLFlow and the client are running on the same host or they map a directory that leads to the same location over NFS, for example.
Some (untested) code along the lines of:
function logartifact(mlf::GitLabMLFlow, run_id::AbstractString, basefilename::AbstractString, data)
mlflowrun = getrun(mlf, run_id)
artifact_uri = mlflowrun.info.artifact_uri
filepath = joinpath(artifact_uri, basefilename)
try
HTTP.post(url, push!(copy(mlf.headers), "Content-Type" => "multipart/form-data" ),
HTTP.Form(Dict(:file => data)))
catch e
error("Unable to upload $(filepath): $e")
end
filepath
end
Would probably work for this purpose and not introduce new dependencies.
The question remains how to differentiate between local and remote file access, as they happen through different interfaces (the headers are irrelevant for file access for example).
I see three options to approach this and would be interested in making a pull request implementing one of these and testing them against a local mlflow instance and the gitlab end points to confirm they are working (for me).
AbstractMLFlow
to denote remote instances. (Probably not preferred)This is simply a question of taste i would leave to (some of) the maintainers.
The MLFlow REST API Create Run and Update Run documentation shows that start_time
and end_time
should be Unix timestamp.
And from Converting Date/Time without time zone information and mlflow/mlflow#30 (comment), we know that Unix timestamp should be UTC+0.
But createrun
and updaterun
are using local time, causing the wrong time to be displayed on MLFlow dashboard.
Lines 17 to 24 in 7c88ec4
Lines 64 to 77 in 7c88ec4
I would like to open a conversion about adding support for remote MLFlow server.
User story: As a corporate employee, I'd like to be able to send MLFlow tracking information to our company server to have experiments across all teams in one place.
Feature needed: Adding basic auth or bearer token to the header of the HTTP requests.
1, Would you be interested in adding it? I'm happy to open a PR.
2, What would be the suggested implementation?
Re 2)
I would propose to:
headers()
method (similar to uri()
method) that would set up the right headers for mlfget()
and mlfpost()
methods in utils.jlWhat do you think?
Hi,
It would be great if there was an example showing how to log model architecture
Is that possible currently ?
It seems one cannot log runs to a single experiment asynchronously:
using MLJModels
using MLJBase
using MLFlowClient
using MLJFlow
logger = MLFlowLogger("http://127.0.0.1:5000", experiment_name="white moon")
X, y = @load_iris
using .Threads
model = (@iload DecisionTreeClassifier pkg=DecisionTree)()
nthreads()
# 5
@sync for i in 1:5
Threads.@spawn evaluate(model, X, y; logger)
end
nested task error: HTTP.Exceptions.StatusError(400, "POST", "/api/2.0/mlflow/experiments/create", HTTP.Messages.Response:
"""
HTTP/1.1 400 Bad Request
Server: gunicorn
Date: Sun, 24 Sep 2023 20:35:24 GMT
Connection: close
Content-Type: application/json
Content-Length: 95
{"error_code": "RESOURCE_ALREADY_EXISTS", "message": "Experiment 'white moon' already exists."}""")
Stacktrace:
[1] mlfpost(mlf::MLFlow, endpoint::String; kwargs::Base.Pairs{Symbol, Union{Missing, Nothing, String}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:name, :artifact_location, :tags), Tuple{String, Nothing, Missing}}})
@ MLFlowClient ~/.julia/packages/MLFlowClient/Szkbv/src/utils.jl:74
[2] mlfpost
@ ~/.julia/packages/MLFlowClient/Szkbv/src/utils.jl:66 [inlined]
[3] createexperiment(mlf::MLFlow; name::String, artifact_location::Nothing, tags::Missing)
@ MLFlowClient ~/.julia/packages/MLFlowClient/Szkbv/src/experiments.jl:21
[4] createexperiment
@ ~/.julia/packages/MLFlowClient/Szkbv/src/experiments.jl:16 [inlined]
[5] #getorcreateexperiment#7
@ ~/.julia/packages/MLFlowClient/Szkbv/src/experiments.jl:103 [inlined]
[6] log_evaluation(logger::MLFlowLogger, performance_evaluation::PerformanceEvaluation
{MLJDecisionTreeInterface.DecisionTreeClassifier, Vector{LogLoss{Float64}}, Vector{Float64}, Vector{typeof(predict)}, Vector{Vector{Float64}}, Vector{Vector{Vector{Float64}}}, Vector{NamedTuple{(:tree, :raw_tree, :encoding, :features), Tuple{DecisionTree.InfoNode{Float64, UInt32}, DecisionTree.Root{Float64, UInt32}, Dict{UInt32, CategoricalArrays.CategoricalValue{String, UInt32}}, Vector{Symbol}}}}, Vector{NamedTuple{(:classes_seen, :print_tree, :features), Tuple{CategoricalArrays.CategoricalVector{String, UInt32, String, CategoricalArrays.CategoricalValue{String, UInt32}, Union{}}, MLJDecisionTreeInterface.TreePrinter{DecisionTree.Root{Float64, UInt32}}, Vector{Symbol}}}}, CV})
@ MLJFlow ~/.julia/packages/MLJFlow/TqEtw/src/base.jl:2
[7] evaluate!(mach::Machine{MLJDecisionTreeInterface.DecisionTreeClassifier, true}, resampling::Vector{Tuple{Vector{Int64}, UnitRange{Int64}}}, weights::Nothing, class_weights::Nothing, rows::Nothing, verbosity::Int64, repeats::Int64, measures::Vector{LogLoss{Float64}}, operations::Vector{typeof(predict)}, acceleration::CPU1{Nothing}, force::Bool, logger::MLFlowLogger, user_resampling::CV)
@ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1314
[8] evaluate!(::Machine{MLJDecisionTreeInterface.DecisionTreeClassifier, true}, ::CV, ::Nothing, ::Nothing, ::Nothing, ::Int64, ::Int64, ::Vector{LogLoss{Float64}}, ::Vector{typeof(predict)}, ::CPU1{Nothing}, ::Bool, ::MLFlowLogger, ::CV)
@ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1335
[9] evaluate!(mach::Machine{MLJDecisionTreeInterface.DecisionTreeClassifier, true}; resampling::CV, measures::Nothing, measure::Nothing, weights::Nothing, class_weights::Nothing, operations::Nothing, operation::Nothing, acceleration::CPU1{Nothing}, rows::Nothing, repeats::Int64, force::Bool, check_measure::Bool, verbosity::Int64, logger::MLFlowLogger)
@ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1015
[10] evaluate(::MLJDecisionTreeInterface.DecisionTreeClassifier, ::NamedTuple{(:sepal_length, :sepal_width, :petal_length, :petal_width), NTuple{4, Vector{Float64}}}, ::Vararg{Any}; cache::Bool, kwargs::Base.Pairs{Symbol, MLFlowLogger, Tuple{Symbol}, NamedTuple{(:logger,), Tuple{MLFlowLogger}}})
@ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1029
[11] (::var"#7#8")()
@ Main ./threadingconstructs.jl:373
<repeats several times>
I got it sort of working in GitLab
For that i needed to undo #36 locally.
And artifact uploading is currently broken due to it not being supported by the library. I have a patch for that too.
I see two paths forward i could see working:
MLFlow
to allow different endpoints based on arguments passed at the MLFlow struct creation(MLFlowClient) pkg> status
Project MLFlowClient v0.4.6
Status `~/.julia/dev/MLFlowClient/Project.toml`
[48062228] FilePathsBase v0.9.21
[cd3eb016] HTTP v1.10.5
[682c06a0] JSON v0.21.4
[605ecd9f] ShowCases v0.1.0
[5c2747f8] URIs v1.5.1
[ade2ca70] Dates
[cf7118a7] UUIDs
(MLFlowClient) pkg> test
Testing MLFlowClient
Status `/tmp/jl_1EBWsE/Project.toml`
[48062228] FilePathsBase v0.9.21
[cd3eb016] HTTP v1.10.5
[682c06a0] JSON v0.21.4
[64a0f543] MLFlowClient v0.4.6 `~/.julia/dev/MLFlowClient`
[605ecd9f] ShowCases v0.1.0
[5c2747f8] URIs v1.5.1
[ade2ca70] Dates
[8dfed614] Test
[cf7118a7] UUIDs
Status `/tmp/jl_1EBWsE/Manifest.toml`
[d1d4a3ce] BitFlags v0.1.8
[944b1d66] CodecZlib v0.7.4
[34da2185] Compat v4.14.0
[f0e56b4a] ConcurrentUtilities v2.4.1
[460bff9d] ExceptionUnwrapping v0.1.10
[48062228] FilePathsBase v0.9.21
[cd3eb016] HTTP v1.10.5
[692b3bcd] JLLWrappers v1.5.0
[682c06a0] JSON v0.21.4
[e6f89c97] LoggingExtras v1.0.3
[64a0f543] MLFlowClient v0.4.6 `~/.julia/dev/MLFlowClient`
[739be429] MbedTLS v1.1.9
[4d8831e6] OpenSSL v1.4.2
[69de0a69] Parsers v2.8.1
[aea7be01] PrecompileTools v1.2.1
[21216c6a] Preferences v1.4.3
[605ecd9f] ShowCases v0.1.0
[777ac1f9] SimpleBufferStream v1.1.0
[3bb67fe8] TranscodingStreams v0.10.7
[5c2747f8] URIs v1.5.1
[458c3c95] OpenSSL_jll v3.0.13+1
[56f22d72] Artifacts
[2a0f44e3] Base64
[ade2ca70] Dates
[b77e0a4c] InteractiveUtils
[8f399da3] Libdl
[56ddb016] Logging
[d6f4376e] Markdown
[a63ad114] Mmap
[ca575930] NetworkOptions v1.2.0
[de0858da] Printf
[9a3f8284] Random
[ea8e919c] SHA v0.7.0
[9e88b42a] Serialization
[6462fe0b] Sockets
[fa267f1f] TOML v1.0.3
[8dfed614] Test
[cf7118a7] UUIDs
[4ec0a83e] Unicode
[c8ffd9c3] MbedTLS_jll v2.28.2+0
[14a3606d] MozillaCACerts_jll v2022.10.11
[83775a58] Zlib_jll v1.2.13+0
Testing Running tests...
Test Summary: | Pass Total Time
createexperiment | 2 2 0.6s
Test Summary: | Pass Total Time
getexperiment | 5 5 0.1s
getexperiment_by_experiment_id | 2 2 0.0s
getexperiment_by_experiment_name | 2 2 0.1s
getexperiment_not_found | 1 1 0.0s
Test Summary: | Pass Total Time
getorcreateexperiment | 4 4 0.1s
deleteexperiment: Test Failed at /home/anabrid/.julia/dev/MLFlowClient/test/test_experiments.jl:53
Expression: length(experiments) == 1
Evaluated: 2 == 1
Stacktrace:
[1] macro expansion
@ ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Test/src/Test.jl:478 [inlined]
[2] macro expansion
@ ~/.julia/dev/MLFlowClient/test/test_experiments.jl:53 [inlined]
[3] macro expansion
@ ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[4] top-level scope
@ ~/.julia/dev/MLFlowClient/test/test_experiments.jl:48
Test Summary: | Fail Total Time
deleteexperiment | 1 1 0.7s
ERROR: LoadError: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /home/anabrid/.julia/dev/MLFlowClient/test/test_experiments.jl:47
in expression starting at /home/anabrid/.julia/dev/MLFlowClient/test/runtests.jl:3
ERROR: Package MLFlowClient errored during testing
shell> git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
shell> git show HEAD
commit a31b41a91dc32107fa7dcd6ad5a581c116a4ff9c (HEAD -> main, origin/main, origin/gitlab_client, origin/HEAD)
Author: Jose Esparza <[email protected]>
Date: Tue Mar 5 11:23:19 2024 -0500
Bump version
diff --git a/Project.toml b/Project.toml
index 6c4ebe8..202535a 100644
--- a/Project.toml
+++ b/Project.toml
@@ -1,7 +1,7 @@
name = "MLFlowClient"
uuid = "64a0f543-368b-4a9a-827a-e71edb2a0b83"
authors = ["@deyandyankov, @pebeto, and contributors"]
-version = "0.4.5"
+version = "0.4.6"
[deps]
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
shell> git log
commit a31b41a91dc32107fa7dcd6ad5a581c116a4ff9c (HEAD -> main, origin/main, origin/gitlab_client, origin/HEAD)
Author: Jose Esparza <[email protected]>
Date: Tue Mar 5 11:23:19 2024 -0500
Bump version
commit 776695b6bb4d92d732b162ae92de72b8ebbf4aba
Merge: de98ac2 60a2535
Author: Jose Esparza <[email protected]>
Date: Mon Mar 4 18:14:18 2024 -0500
Merge pull request #37 from pebeto/improving_code_base
Improving code base
commit 60a25356276ab61aed12eba7af45b944adc85d44
Author: Jose Esparza <[email protected]>
Date: Mon Mar 4 18:11:11 2024 -0500
Ignoring `deprecated.jl` in coverage
commit dfed176c092e0194d6e220434cfd06a7d346b41c
Author: Jose Esparza <[email protected]>
Date: Sun Jan 7 23:29:12 2024 -0500
Increasing coverage
commit 3190751aef412958151d509d8866c8877636ba38
Author: Jose Esparza <[email protected]>
Date: Sun Jan 7 23:13:06 2024 -0500
Fixing Documenter failing on pipeline
commit 8886aa835e9fb176e1c774e36e69593c983c440d
Author: Jose Esparza <[email protected]>
Date: Sun Jan 7 22:22:54 2024 -0500
Increasing maintainability by nice function separation in files
commit de98ac2fddbd49e00a3724d05184d1462f8a291c
Author: deyandyankov <[email protected]>
Date: Sun Jan 7 21:58:13 2024 +0000
@JuliaRegistrator register
commit 26674e04bf1313942d78f682924e6683ac6106c7
Author: deyandyankov <[email protected]>
Date: Sun Jan 7 21:54:10 2024 +0000
bump version
commit e27ac1026627327d58fd4b63aeeef62c738b0977
Author: Jose Esparza <[email protected]>
Date: Sun Jan 7 16:53:07 2024 -0500
Migrating from /api to /ajax-api endpoint (#36)
commit 5a4237fbd28a611c40d37b6cc705a3c0b2de2c50
Author: deyandyankov <[email protected]>
Date: Thu Aug 10 09:54:40 2023 +0100
Hi,
Current version of MLFlow is showing a special attributes for a run where dataset used and model is indicated.
Would be good to include possibility of using that. Currently I am storing dataset in parameters.
Referencing JuliaAI/MLJFlow.jl#42
Hello,
I have trouble setting tags for an experiment.
On the code
###
#header
###
mlf = MLFlow("http://localhost:5000//api")
experiment_name = "tag_run"
experiment_tags = Dict("tag1" => "tag1value")
createexperiment(mlf; name=experiment_name, tags = experiment_tags)
I get a HTTP 400 error :
HTTP/1.1 400 Bad Request
Server: gunicorn
Date: Tue, 04 Jun 2024 10:26:14 GMT
Connection: close
Content-Type: application/json
Content-Length: 218
while
###
#header
###
mlf = MLFlow("http://localhost:5000//api")
experiment_name = "tag_run"
createexperiment(mlf; name=experiment_name)
runs as expected.
I tried to understand the problem a little bit but I don't think I quite understand it.
The createexperiment function parses the Dictionary as an argument to mlfpost which itself creates a body = JSON.json(kwargs)
So the HTTP request should read something like:
HTTP.post(apiuri, apiheaders, body) = HTTP.post( apiuri, apiheads, "{\"tags\":{\"tag1\":\"tag1value\"}}" )
Hi,
The example simple-with-mlflow.jl do not work anymore with v0.5.1 :
ERROR: LoadError: HTTP.ConnectError for url = `http://localhost:5000/2.0/mlflow/experiments/create`: IOError: connect: connection refused (ECONNREFUSED)
But it's fine with v0.4.7.
Don't understand why, for now, I downgrade to v0.4.7.
Looking forward to hearing back from you, I wish you good luck.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.