Comments (6)
@visr, thanks for opening the issue -
- Windows support would be a complex undertaking (given our liberal use of multiprocessing for starters). We are definitely open to contributions and a good way to kick start the conversation would be to do a POC.
- We are updating our contributing guidelines to reflect this. Expect some progress on that front in the next couple of days.
from metaflow.
@savingoyal re:
Windows support would be a complex undertaking (given our liberal use of multiprocessing for starters). We are definitely open to contributions and a good way to kick start the conversation would be to do a POC.
FWIW, all data scientists may not be running a *nix Linux or mac. Having official support for Windows would be a big plus. I wish I could coerce them though but that's a dream. (And FWIW, WSL or Cygwin are their own can of frankenstein worms)
Multiprocessing may not be the main issue IMHO but rather:
-
keeping Python 2 and 3 compatibility is going to be a major pain on Windows
-
using signals such as here https://github.com/Netflix/metaflow/blob/c2f70c73782549fdce6c418410ecd9b305199b28/metaflow/plugins/timeout_decorator.py
-
possibly some assumptions about calling some functions and file system semantics that do not apply or are not available on Windows
For 2. Windows is a pain but I have some POSIX/windows utilities that have been battle tested for timeouts here https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/interrupt.py
For 1. we did that in https://github.com/nexB/scancode-toolkit and I would never do it again which would mean to support Python 3 for Windows. Anything else is a sure way to destroy grey matter very quickly ;)
from metaflow.
And what about news about Windows supporting Linux subsystem?
from metaflow.
My latest thinking is that Windows support is too mucho pita and with the new WSL2, running on Windows means running on Linux, which is a sane thing. I would close this once some doc is added that WSL2 must be used
from metaflow.
I was hoping that a full remote k8s execution of a flow would work on windows but it doesn't. I stubbed out fcntl but hit this polling dependency. I am in a corporate setup so windows is unfortunately my client. I could develop flows locally in docker but it would be nice to be able to execute the flow in k8s from windows.
$ python 00-helloworld/helloworld.py run --with kubernetes:service_account=auser
Metaflow 2.5.2.post7+git6b07668 executing HelloFlow for user:U038178
Validating your flow...
The graph looks good!
Running pylint...
Pylint is happy!
Polling is not supported on your operating system (Windows)
2022-03-17 13:23:24.242 Workflow starting (run-id 1647523400851411):
2022-03-17 13:23:24.628 [1647523400851411/start/1 (pid 34040)] Task is starting.
2022-03-17 13:23:24.629 Workflow failed.
2022-03-17 13:23:24.630 Terminating 1 active tasks...
2022-03-17 13:23:24.631 [1647523400851411/start/1 (pid 34040)] [KILLED BY ORCHESTRATOR]
2022-03-17 13:23:24.632 [1647523400851411/start/1 (pid 34040)] [KILLED BY ORCHESTRATOR]
2022-03-17 13:23:29.000 Killing 1 remaining tasks after having waited for 5 seconds -- some tasks may not exit clean
2022-03-17 13:23:29.002 Flushing logs...
Internal error
Traceback (most recent call last):
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 209, in execute
self._launch_workers()
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 564, in _launch_workers
self._launch_worker(task)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 584, in _launch_worker
self._poll.add(fd)
AttributeError: 'NoneType' object has no attribute 'add'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 1102, in main
start(auto_envvar_prefix="METAFLOW", obj=state)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 829, in __call__
return self.main(args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 610, in invoke
return callback(args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 666, in wrapper
return func(args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\decorators.py", line 33, in new_func
return f(get_current_context().obj, args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 789, in run
runtime.execute()
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 230, in execute
self._killall()
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 275, in _killall
list(self._poll_workers())
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 521, in _poll_workers
for event in self._poll.poll(PROGRESS_INTERVAL):
AttributeError: 'NoneType' object has no attribute 'poll'
I see Argo support might be forthcoming #50 would this be an option for starting flows in k8s from windows or is windows as a client completely out?
Also its not clear to me how the code is packaged? I see decorators for conda & image and reference to the code being tarball'd to S3. Can you explain or point me to the docs that explain the code packaging for k8s. Thanks.
from metaflow.
@erowan Would you like to try out #992 in your windows environment? Here is the code that packages up your flow.
from metaflow.
Related Issues (20)
- Missing artifact in the step.task.data._artifacts HOT 1
- Make default kubernetes resources configurable HOT 2
- `python hello.py batch step --help` is broken
- Invalid CloudWatch log group prefix HOT 6
- Allow namespacing of `ArgoEvent` when published from a step
- allow uploading metaflow packages that include "dots" in the path
- Suggest replacing `pull_request_target` in branches as well as `main`
- Bug: Passing an `S3PutObject` to `s3.put_files` treated as `tuple` of key path values.
- Allow passing of `trusted-host` parameter to `@pypi` decorator.
- Unable to utilise pytest unit test for metaflow HOT 1
- Update extras_require for tracing dependencies?
- R tests on Github fail with macos-latest runner
- Argo-workflows with name > 52 characters fail if run as CronWorkflows HOT 1
- Introduce support for private registry for AWS Batch
- Write metadata about log scrubbing events
- Language-agnostic API and/or other language SDKs for the client API HOT 1
- Performance Degradation When Running PyCaret Model with n_jobs > 1 Inside Metaflow
- Question on Executing Metaflow Workflow from Python Script Without 'run' Argument HOT 3
- Resume Doesn't Resume from the Last Failed Step
- Argo Events: trigger sensors are not deleted
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metaflow.