konchunas / pyrs Goto Github PK
View Code? Open in Web Editor NEWPython to Rust transpiler
License: MIT License
Python to Rust transpiler
License: MIT License
I suggest doing what bindgen
does and just adding an underscore:
struct -> struct_
type -> type_
setup.py uses name 'pyrs', but https://pypi.org/project/pyrs/ is already taken.
python-rs
is available.
It would be great to see this on PyPI, even in alpha/pre-release status.
Would you be open for an setup.py contribution to allow your project to be treated better in system package manager context?
Instead of full blown type inference (which will be VERY hard to do because of python's loose, dynamic typing), why not make the resulting code generate generics, and then halt the transpiling if it sees type changing?
I can help implement. If you need, let me know.
I want to generate something like this:
#[derive(BorshSerialize, BorshDeserialize, Debug)]
pub struct GreetingAccount {
/// number of greetings
pub counter: u32,
}
Is there a way to generate the derive statements automatically from Python code?
Thanks!
It looks like pyrs does not handle strings like
new_lines = '\r\n'
or "\"
Very well, it seems to interpret them into the escaped character, which rust doesn't appreciate as it also expects them to be escaped.
Though I can use rust-fmt, it is still not good.
I tried transpiling a python script today and ran into this:
Traceback (most recent call last):
File "./pyrs.py", line 20, in
rs = transpile(source)
File "/home/egil/src/rust/python2rust/pyrs/transpiler.py", line 34, in transpile
return transpiler.visit(tree)
File "/home/egil/src/rust/python2rust/pyrs/clike.py", line 74, in visit
return super(CLikeTranspiler, self).visit(node)
File "/usr/lib/python3.8/ast.py", line 363, in visit
return visitor(node)
File "/home/egil/src/rust/python2rust/pyrs/transpiler.py", line 280, in visit_Module
buf += [self.visit(b) for b in node.body]
File "/home/egil/src/rust/python2rust/pyrs/transpiler.py", line 280, in
buf += [self.visit(b) for b in node.body]
File "/home/egil/src/rust/python2rust/pyrs/clike.py", line 74, in visit
return super(CLikeTranspiler, self).visit(node)
File "/usr/lib/python3.8/ast.py", line 363, in visit
return visitor(node)
File "/home/egil/src/rust/python2rust/pyrs/transpiler.py", line 193, in visit_Expr
s = self.visit(node.value)
File "/home/egil/src/rust/python2rust/pyrs/clike.py", line 74, in visit
return super(CLikeTranspiler, self).visit(node)
File "/usr/lib/python3.8/ast.py", line 363, in visit
return visitor(node)
File "/home/egil/src/rust/python2rust/pyrs/transpiler.py", line 133, in visit_Call
args = ", ".join(args)
TypeError: sequence item 0: expected str instance, NoneType found
source: https://gitlab.com/-/snippets/2001904
$ cd pyrs/pyrs; pytest-3
Gives me:
6 failed, 27 passed
What do you think about restoring the py14 code and the tests, so we can have one tree where both C++ and Rust are functional along with passing tests?
I'm trying to generate a function like the following in Rust:
use my_program::{MyImportedClass, AccountInfo, ProgramResult};
pub fn process_instruction(
program_id: &MyImportedClass,
accounts: &[AccountInfo]
) -> ProgramResult {
/* pass */
Is there a way to currently generate this? Many thanks
I just wanted to propose that decorators are resolved like this:
@{decorator}
def {func_name}({args}):
{code}
to
let {func_name} = {decorator}(|| {
{code}
})
Unfortunately, I am not used to Rust, so this might be totally wrong, but I saw that decorators weren't implemented yet and thought this might be a simple improvement
I couldn't find a better way to contact @konchunas :P
Anyway, heads-up. 6 months ago I was trying to measure an algorithm written in Python and NumPy and ended up transpiling it to Rust to get rid of GIL and get readable numbers with perf.
Stuff was successful and I ended up writing a paper on evaluating the feasibility of transpiling Python to Rust. pyrs is featured rather prominently. I also found a collection of issues and possible solutions in the process.
Paper is to be reviewed during the spring and if accepted, to be published in July. Let me know if you'd like an elaboration apart from the paper.
File "/home/db48x/projects/pyrs/pyrs/clike.py", line 79, in visit_Return
return 'return {0};'.format(self.visit(node.value))
File "/home/db48x/projects/pyrs/pyrs/clike.py", line 52, in visit
return super(CLikeTranspiler, self).visit(node)
File "/usr/lib64/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/home/db48x/projects/pyrs/pyrs/transpiler.py", line 359, in visit_Tuple
elts = ", ".join(elts)
Looks like the code is this line, so I don't know why the second element is None:
return (output, errcode[0] if len(errcode) > 0 else 0)
Hello, i would like to show the results i obtained using pyrs together with monkeytype and the process i followed.
These are the first commands i used:
cd typing
echo 'layout python3' > .envrc
direnv allow
git clone https://github.com/chonyy/fpgrowth_py.git
git clone https://github.com/Instagram/MonkeyType.git
git clone https://github.com/konchunas/pyrs.git
cd MonkeyType
python3 -m pip install -e .
cd ../fpgrowth_py
monkeytype run run.py
monkeytype list-modules # optional step that shows what `apply` accepts
monkeytype apply fpgrowth_py.utils
git clone https://github.com/chonyy/fpgrowth_py.git
is here the project that is being converted but it might be any project.
After that i ran
monkeytype apply fpgrowth_py.fpgrowth
But i ran into a problem
File "/home/flip111/typing/fpgrowth_py/fpgrowth_py/utils.py", line 7, in Node
def __init__(self, itemName: str, frequency: int, parentNode: Optional[Node]) -> None:
NameError: name 'Node' is not defined
I solved this by temporarily removing the type definition part : Optional[Node]
, then running monkeytype again and then putting the type definition back.
After that it was time for pyrs
python3 -m pyrs fpgrowth_py/fpgrowth_py/fpgrowth.py > fpgrowth_py/fpgrowth_py/fpgrowth.rs
python3 -m pyrs fpgrowth_py/fpgrowth_py/utils.py > fpgrowth_py/fpgrowth_py/utils.rs
rustfmt fpgrowth_py/fpgrowth_py/fpgrowth.rs
rustfmt fpgrowth_py/fpgrowth_py/utils.rs
rustfmt
then complains
» rustfmt fpgrowth_py/fpgrowth_py/utils.rs 6 files, 435 ins.(+), 268 del.(-) [14:48:38]
error: unexpected closing delimiter: `}`
--> /home/flip111/typing/fpgrowth_py/fpgrowth_py/utils.rs:48:1
|
34 | fn getFromFile<T0, RT>(fname: T0) -> RT {
| - this opening brace...
...
46 | }
| - ...matches this closing brace
47 | return (itemSetList, frequency);
48 | }
| ^ unexpected closing delimiter
Because python source code
def getFromFile(fname):
itemSetList = []
frequency = []
with open(fname, 'r') as file:
csv_reader = reader(file)
for line in csv_reader:
line = list(filter(None, line))
itemSetList.append(line)
frequency.append(1)
return itemSetList, frequency
got translated into (i indented this for convenience of reading this post)
fn getFromFile<T0, RT>(fname: T0) -> RT {
let mut itemSetList = vec![];
let mut frequency = vec![];
// with!(open(fname, "r") as file) //unsupported
{
let csv_reader = reader(file);
}
for line in csv_reader {
line = line.into_iter().filter(None).collect::<Vec<_>>();
itemSetList.push(line);
frequency.push(1);
}
}
return (itemSetList, frequency);
}
There is the unsupported with
language construct together with a file open. Also one closing bracket got introduced after let csv_reader = reader(file);
for some reason. I manually fixed this into
fn getFromFile<T0, RT>(fname: T0) -> RT {
let mut itemSetList = vec![];
let mut frequency = vec![];
// with!(open(fname, "r") as file) //unsupported
if let Ok(file) = std::fs::File::open(fname) {
let csv_reader = reader(file);
for line in csv_reader {
line = line.into_iter().filter(None).collect::<Vec<_>>();
itemSetList.push(line);
frequency.push(1);
}
}
return (itemSetList, frequency);
}
I previously ran this process without the monkeytype step. After this i was able to make a diff of the resulting rust source code. Here is a diff of the pygrowth.rs
file which shows also in Rust there are a lot more concrete types available
9,11c9,11
< itemName: ST0,
< count: ST1,
< parent: ST2,
---
> itemName: &str,
> count: i32,
> parent: Option<Node>,
17c17
< fn __init__<T0, T1, T2>(&self, itemName: T0, frequency: T1, parentNode: T2) {
---
> fn __init__(&self, itemName: &str, frequency: i32, parentNode: Option<Node>) {
24c24
< fn increment<T0>(&self, frequency: T0) {
---
> fn increment(&self, frequency: i32) {
54c54,58
< fn constructTree<T0, T1, T2, RT>(itemSetList: T0, frequency: T1, minSup: T2) -> RT {
---
> fn constructTree(
> itemSetList: Vec<Union<Any, Vec<&str>>>,
> frequency: Vec<Union<Any, i32>>,
> minSup: f32,
> ) -> Union<(None, None), (Node, HashMap<&str, Vec<Union<i32, Node>>>)> {
92c96,103
< fn updateHeaderTable<T0, T1, T2>(item: T0, targetNode: T1, headerTable: T2) {
---
> fn updateHeaderTable(
> item: &str,
> targetNode: Node,
> headerTable: HashMap<
> &str,
> Union<Vec<Option<i32>>, Vec<Option<Union<i32, Node>>>, Vec<Union<i32, Node>>>,
> >,
> ) {
103c114,122
< fn updateTree<T0, T1, T2, T3, RT>(item: T0, treeNode: T1, headerTable: T2, frequency: T3) -> RT {
---
> fn updateTree(
> item: &str,
> treeNode: Node,
> headerTable: HashMap<
> &str,
> Union<Vec<Option<i32>>, Vec<Option<Union<i32, Node>>>, Vec<Union<i32, Node>>>,
> >,
> frequency: i32,
> ) -> Node {
113c132
< fn ascendFPtree<T0, T1>(node: T0, prefixPath: T1) {
---
> fn ascendFPtree(node: Node, prefixPath: Vec<Union<Any, &str>>) {
119c138,141
< fn findPrefixPath<T0, T1, RT>(basePat: T0, headerTable: T1) -> RT {
---
> fn findPrefixPath(
> basePat: &str,
> headerTable: HashMap<&str, Vec<Union<i32, Node>>>,
> ) -> Union<(Vec<Any>, Vec<Any>), (Vec<Vec<&str>>, Vec<i32>)> {
134c156,161
< fn mineTree<T0, T1, T2, T3>(headerTable: T0, minSup: T1, preFix: T2, freqItemList: T3) {
---
> fn mineTree(
> headerTable: HashMap<&str, Vec<Union<i32, Node>>>,
> minSup: f32,
> preFix: Set<&str>,
> freqItemList: Vec<Union<Set<&str>, Any>>,
> ) {
151c178
< fn powerset<T0, RT>(s: T0) -> RT {
---
> fn powerset(s: Set<&str>) -> chain {
159c186
< fn getSupport<T0, T1, RT>(testSet: T0, itemSetList: T1) -> RT {
---
> fn getSupport(testSet: Union<Set<&str>, (&str)>, itemSetList: Vec<Vec<&str>>) -> i32 {
168c195,199
< fn associationRule<T0, T1, T2, RT>(freqItemSet: T0, itemSetList: T1, minConf: T2) -> RT {
---
> fn associationRule(
> freqItemSet: Vec<Set<&str>>,
> itemSetList: Vec<Vec<&str>>,
> minConf: f32,
> ) -> Vec<Vec<Union<Set<&str>, f32>>> {
182c213
< fn getFrequencyFromList<T0, RT>(itemSetList: T0) -> RT {
---
> fn getFrequencyFromList(itemSetList: Vec<Vec<&str>>) -> Vec<i32> {
After this i copied the two new source files into a new project
cargo new fpgrowth_rs
cp fpgrowth_py/fpgrowth_py/fpgrowth.rs fpgrowth_rs/src
cp fpgrowth_py/fpgrowth_py/utils.rs fpgrowth_rs/src
cd fpgrowth_rs
I added an import for fpgrowth into src/main.rs
mod fpgrowth;
fn main() {
println!("Hello, world!");
}
I then tried to fix the source files with clippy
cargo clippy --fix --allow-dirty
Clippy reported the following errors
» cargo clippy --fix --allow-dirty
Checking fpgrowth_rs v0.1.0 (/home/flip111/typing/fpgrowth_rs)
error[E0433]: failed to resolve: use of undeclared crate or module `fpgrowth_py`
--> src/fpgrowth.rs:6:5
|
6 | use fpgrowth_py::utils::*;
| ^^^^^^^^^^^ use of undeclared crate or module `fpgrowth_py`
|
help: there is a crate or module with a similar name
|
6 | use fpgrowth::utils::*;
| ~~~~~~~~
error[E0432]: unresolved imports `collections::defaultdict`, `collections::OrderedDict`
--> src/fpgrowth.rs:4:19
|
4 | use collections::{defaultdict, OrderedDict};
| ^^^^^^^^^^^ ^^^^^^^^^^^ no `OrderedDict` in `collections`
| |
| no `defaultdict` in `collections`
error[E0432]: unresolved import `csv`
--> src/fpgrowth.rs:5:5
|
5 | use csv::reader;
| ^^^ use of undeclared crate or module `csv`
error[E0432]: unresolved import `itertools`
--> src/fpgrowth.rs:7:5
|
7 | use itertools::{chain, combinations};
| ^^^^^^^^^ use of undeclared crate or module `itertools`
error[E0432]: unresolved import `optparse`
--> src/fpgrowth.rs:8:5
|
8 | use optparse::OptionParser;
| ^^^^^^^^ use of undeclared crate or module `optparse`
error[E0412]: cannot find type `Set` in this scope
--> src/fpgrowth.rs:14:11
|
14 | ) -> (Vec<Set<&str>>, Vec<Vec<Union<Set<&str>, f32>>>) {
| ^^^ not found in this scope
error[E0412]: cannot find type `Union` in this scope
--> src/fpgrowth.rs:14:31
|
14 | ) -> (Vec<Set<&str>>, Vec<Vec<Union<Set<&str>, f32>>>) {
| ^^^^^ not found in this scope
|
help: consider importing one of these items
|
1 | use crate::fpgrowth::collections::btree_set::Union;
|
1 | use crate::fpgrowth::collections::hash_set::Union;
|
1 | use std::collections::btree_set::Union;
|
1 | use std::collections::hash_set::Union;
|
error[E0412]: cannot find type `Set` in this scope
--> src/fpgrowth.rs:14:37
|
14 | ) -> (Vec<Set<&str>>, Vec<Vec<Union<Set<&str>, f32>>>) {
| ^^^ not found in this scope
error[E0425]: cannot find function `getFrequencyFromList` in this scope
--> src/fpgrowth.rs:15:21
|
15 | let frequency = getFrequencyFromList(itemSetList);
| ^^^^^^^^^^^^^^^^^^^^ not found in this scope
error[E0425]: cannot find function `constructTree` in this scope
--> src/fpgrowth.rs:17:33
|
17 | let (fpTree, headerTable) = constructTree(itemSetList, frequency, minSup);
| ^^^^^^^^^^^^^ not found in this scope
error[E0425]: cannot find function `mineTree` in this scope
--> src/fpgrowth.rs:22:9
|
22 | mineTree(headerTable, minSup, set(), freqItems);
| ^^^^^^^^ not found in this scope
error[E0425]: cannot find function `set` in this scope
--> src/fpgrowth.rs:22:39
|
22 | mineTree(headerTable, minSup, set(), freqItems);
| ^^^ not found in this scope
error[E0425]: cannot find function `associationRule` in this scope
--> src/fpgrowth.rs:23:21
|
23 | let rules = associationRule(freqItems, itemSetList, minConf);
| ^^^^^^^^^^^^^^^ not found in this scope
error[E0425]: cannot find function `getFromFile` in this scope
--> src/fpgrowth.rs:28:36
|
28 | let (itemSetList, frequency) = getFromFile(fname);
| ^^^^^^^^^^^ not found in this scope
error[E0425]: cannot find function `constructTree` in this scope
--> src/fpgrowth.rs:30:33
|
30 | let (fpTree, headerTable) = constructTree(itemSetList, frequency, minSup);
| ^^^^^^^^^^^^^ not found in this scope
error[E0425]: cannot find function `mineTree` in this scope
--> src/fpgrowth.rs:35:9
|
35 | mineTree(headerTable, minSup, set(), freqItems);
| ^^^^^^^^ not found in this scope
error[E0425]: cannot find function `set` in this scope
--> src/fpgrowth.rs:35:39
|
35 | mineTree(headerTable, minSup, set(), freqItems);
| ^^^ not found in this scope
error[E0425]: cannot find function `associationRule` in this scope
--> src/fpgrowth.rs:36:21
|
36 | let rules = associationRule(freqItems, itemSetList, minConf);
| ^^^^^^^^^^^^^^^ not found in this scope
warning: unused import: `std::collections::HashMap`
--> src/fpgrowth.rs:1:5
|
1 | use std::collections::HashMap;
| ^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
warning: unnecessary parentheses around assigned value
--> src/fpgrowth.rs:16:18
|
16 | let minSup = (itemSetList.len() * minSupRatio);
| ^ ^
|
= note: `#[warn(unused_parens)]` on by default
help: remove these parentheses
|
16 - let minSup = (itemSetList.len() * minSupRatio);
16 + let minSup = itemSetList.len() * minSupRatio;
|
warning: unnecessary parentheses around assigned value
--> src/fpgrowth.rs:29:18
|
29 | let minSup = (itemSetList.len() * minSupRatio);
| ^ ^
|
help: remove these parentheses
|
29 - let minSup = (itemSetList.len() * minSupRatio);
29 + let minSup = itemSetList.len() * minSupRatio;
|
error[E0277]: cannot multiply `usize` by `f32`
--> src/fpgrowth.rs:16:37
|
16 | let minSup = (itemSetList.len() * minSupRatio);
| ^ no implementation for `usize * f32`
|
= help: the trait `std::ops::Mul<f32>` is not implemented for `usize`
error[E0308]: mismatched types
--> src/fpgrowth.rs:31:23
|
27 | fn fpgrowthFromFile<T0, T1, T2, RT>(fname: T0, minSupRatio: T1, minConf: T2) -> RT {
| -- this type parameter
...
31 | if fpTree == None {
| _______________________^
32 | | println!("{:?} ", "No frequent item set");
33 | | } else {
| |_____^ expected type parameter `RT`, found `()`
|
= note: expected type parameter `RT`
found unit type `()`
error[E0308]: mismatched types
--> src/fpgrowth.rs:37:16
|
27 | fn fpgrowthFromFile<T0, T1, T2, RT>(fname: T0, minSupRatio: T1, minConf: T2) -> RT {
| -- this type parameter -- expected `RT` because of return type
...
37 | return (freqItems, rules);
| ^^^^^^^^^^^^^^^^^^ expected type parameter `RT`, found tuple
|
= note: expected type parameter `RT`
found tuple `(std::vec::Vec<_>, _)`
Some errors have detailed explanations: E0277, E0308, E0412, E0425, E0432, E0433.
For more information about an error, try `rustc --explain E0277`.
warning: `fpgrowth_rs` (bin "fpgrowth_rs" test) generated 3 warnings
error: could not compile `fpgrowth_rs` due to 21 previous errors; 3 warnings emitted
warning: build failed, waiting for other jobs to finish...
warning: `fpgrowth_rs` (bin "fpgrowth_rs") generated 3 warnings (3 duplicates)
error: build failed
I have yet to inspect these errors and figure out whether they best be fixed before or after using pyrs
Conclusion:
itertools
don't yet get translated to their rust equivelantTraceback (most recent call last):
File "../pyrs/pyrs.py", line 20, in <module>
rs = transpile(source)
File "/home/db48x/projects/pyrs/pyrs/transpiler.py", line 34, in transpile
return transpiler.visit(tree)
File "/home/db48x/projects/pyrs/pyrs/clike.py", line 52, in visit
return super(CLikeTranspiler, self).visit(node)
File "/usr/lib64/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/home/db48x/projects/pyrs/pyrs/transpiler.py", line 272, in visit_Module
buf += [self.visit(b) for b in node.body]
File "/home/db48x/projects/pyrs/pyrs/transpiler.py", line 272, in <listcomp>
buf += [self.visit(b) for b in node.body]
File "/home/db48x/projects/pyrs/pyrs/clike.py", line 52, in visit
return super(CLikeTranspiler, self).visit(node)
File "/usr/lib64/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/home/db48x/projects/pyrs/pyrs/transpiler.py", line 429, in visit_Assign
elements = [self.visit(e) for e in node.value.elts]
File "/home/db48x/projects/pyrs/pyrs/transpiler.py", line 429, in <listcomp>
elements = [self.visit(e) for e in node.value.elts]
File "/home/db48x/projects/pyrs/pyrs/clike.py", line 52, in visit
return super(CLikeTranspiler, self).visit(node)
File "/usr/lib64/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/home/db48x/projects/pyrs/pyrs/transpiler.py", line 133, in visit_Call
args = ", ".join(args)
TypeError: sequence item 3: expected str instance, NoneType found
The args look like this:
['"bzr"', '".bzr"', '"bzr fast-export --no-plain %(basename)s"', None, None, 'None', 'None', '"bzr fast-import -"', '"bzr checkout"', 'set()', 'None', '"http://bazaar.canonical.com/en/"', '".bzrignore"', '"\n# A simulation of bzr default ignores, generated by reposurgeon.\n*.a\n*.o\n*.py[co]\n*.so\n*.sw[nop]\n*~\n.#*\n[#]*#\n__pycache__\nbzr-orphans\n# Simulated bzr default ignores end here\n"', '(VCS::nobug + VCS::bare_numeric)', '"Requires the bzr-fast-import plugin."']
The source code is this:
VCS(name="bzr",
subdirectory=".bzr",
exporter="bzr fast-export --no-plain %(basename)s",
styleflags={"export-progress", "no-nl-after-commit", "nl-after-comment"},
extensions={"empty-directories","multiple-authors","commit-properties"},
initializer=None,
lister=None,
importer="bzr fast-import -",
checkout="bzr checkout",
preserve=set(),
authormap=None,
project="http://bazaar.canonical.com/en/",
ignorename=".bzrignore",
dfltignores="""
# A simulation of bzr default ignores, generated by reposurgeon.
*.a
*.o
*.py[co]
*.so
*.sw[nop]
*~
.#*
[#]*#
__pycache__
bzr-orphans
# Simulated bzr default ignores end here
""",
cookies=(VCS.nobug + VCS.bare_numeric,),
notes="Requires the bzr-fast-import plugin."),
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.