Giter Site home page Giter Site logo

ravro's People

Contributors

piccolbo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ravro's Issues

failing on complicated schemas

Hello,

I tested read.avro using a moderately complicated schema. Some fields contain sub-records, other fields contain arrays of records. One of the sub-records (named moments and containing mean, variance, skewness, and kurtosis fields) is defined the first time and referenced as a type subsequently. This does not cause avro any problems, but read.avro throws the following error:

> dat <- read.avro(file="/path/to/file/part-r-00000.avro")
Error in (function (x, schema, flatten = T, simplify = F, encoded_unions = T,  : 
  Unsupported Avro type: moments

The schema being read here reads (in part):

...
"fields" : [ {
    "name" : "routename",
    "type" : "string",
    "doc" : "path identifier indicates unique fix sequence"
  }, {
    "name" : "aircrafttype",
    "type" : "string"
  }, {
    "name" : "lowaltitudebin",
    "type" : "double",
    "doc" : "altitude [feet] at low end of route (rounded to nearest 1000ft)"
  }, {
    "name" : "highaltitudebin",
    "type" : "double",
    "doc" : "altitude [feet] at high end of route (rounded to nearest 1000ft)"
  }, {
    "name" : "route",
    "type" : [ "null", {
      "type" : "record",
      "name" : "routemetrics",
      "fields" : [ {
        "name" : "route",
        "type" : [ "string", "null" ]
      }, {
        "name" : "initialalttude",
        "type" : [ "null", {
          "type" : "record",
          "name" : "moments",
          "fields" : [ {
            "name" : "mean",
            "type" : "double"
          }, {
            "name" : "variance",
            "type" : "double"
          }, {
            "name" : "skewness",
            "type" : "double"
          }, {
            "name" : "kurtosis",
            "type" : "double"
          }, {
            "name" : "samplesize",
            "type" : "long"
          } ]
        } ],
        "doc" : "moments [feet] characterizing distribution of atltitudes at the beginning of the route (within given binning constraint)"
      }, {
        "name" : "terminalaltitude",
        "type" : [ "null", "moments" ],
        "doc" : "moments [feet] characterizing distribution of atltitudes at the end of the route (within given binning constraint)"
      }, {...

Note that moments is defined as a type (as part of a union) for the first time in the initialalttude field, which is a field of the routemetrics record nested inside of the top-level route field. After that, moments is referenced by name in the subsequent terminalaltitude field.

Are there any plans to deal well with schemas like the one above?

ravro unable to read acro file for my schema

Hello,

I successfully read an avro file from this exemple : https://github.com/miguno/avro-cli-examples/blob/master/twitter.avro but when I try with my schema, a more complex one, ravro gives the following error :

Error in mapply(parse_avro, x_df, schema$fields, MoreArgs = list(flatten = flatten,  :
  zero-length inputs cannot be mixed with those of non-zero length

Here is the schema that I use to create my avro file

{
    "namespace":"com.kiabi.avro.web.orders",
    "type":"record",
    "doc":"This Schema describes about web orders ",
    "name":"Commande",
    "fields": [
        {"name": "metadata",
            "type": {
                "name": "Metadata", "type": "record",
                "fields": [
                    {"name": "date_evenement", "type" : "long", "description": "date d'integration dans la zone de staging - timestamp UNIX"},
                    {"name": "type_evenement", "type" : "string", "description": "Creation, Modication, Suppression"},
                    {"name": "date_integration", "type" : ["null","long"], "description": "date d'integration dans la zone de data - timestamp UNIX"}
                ]
            }
        },
        {"name": "entete_commande",
            "type": {
                "name": "EnteteCommande", "type": "record",
                "fields": [
                    {"name": "code_site_web", "type" : "string"},
                    {"name": "code_cde_web", "type" : "string"},
                    {"name": "no_cli", "type" : "long"},
                    {"name": "no_adresse_cli", "type" : ["null","long"]},
                    {"name": "no_adresse_fact", "type" : ["null","long"]},
                    {"name": "code_etat_cde", "type" : ["null","string"]},
                    {"name": "date_etat_cde", "type" : ["null","long"]},
                    {"name": "code_langue", "type" : ["null","string"]},
                    {"name": "flag_remise_fid", "type" : ["null","string"]},
                    {"name": "flag_dem_carte_fid", "type" : ["null","string"]},
                    {"name": "flag_ach_mag", "type" : ["null","string"]},
                    {"name": "code_orig_connexion", "type" : ["null","long"]},
                    {"name": "no_carte_fid", "type" : ["null","string"]},
                    {"name": "lib_msg_cadeau", "type" : ["null","string"]},
                    {"name": "code_dev", "type" : ["null","string"]},
                    {"name": "code_mode_pay", "type" : ["null","string"]},
                    {"name": "flag_pay_val", "type" : ["null","string"]},
                    {"name": "flag_pay_ann", "type" : ["null","string"]},
                    {"name": "date_pay_ann", "type" : ["null","long"]},
                    {"name": "no_colis_total", "type" : ["null","long"]},
                    {"name": "code_orig_affectation_ca", "type" : ["null","string"]},
                    {"name": "code_etab_affectation_ca", "type" : ["null","string"]},
                    {"name": "code_postal_affectation_ca", "type" : ["null","string"]},
                    {"name": "code_pays_affectation_ca", "type" : ["null","string"]},
                    {"name": "flag_affectation_ca", "type" : ["null","string"]},
                    {"name": "histo_etat_commande",
                        "doc": "on a 1 a N historique de l'etat de la commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "HistoEtatCommande",
                                "fields":[
                                    {"name": "id_histo_etat_cde", "type" : "long"},
                                    {"name": "no_expe", "type" : ["null","string"]},
                                    {"name": "no_reliquat", "type" : ["null","string"]},
                                    {"name": "code_etat_cde", "type" : "string"},
                                    {"name": "date_etat_cde", "type" : "long"}
                                ]
                            }
                        }
                    },
                    {"name": "details_commande",
                        "doc": "on a 1 a N detail par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "DetailsCommande",
                                "fields":[
                                    {"name": "no_ligne", "type" : "long"},
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "no_reliquat", "type" : "string"},
                                    {"name": "code_produit", "type" : "string"},
                                    {"name": "code_type_produit", "type" : "string"},
                                    {"name": "code_orig_cat", "type" : "string"},
                                    {"name": "code_produit_reel", "type" : ["null","string"]},
                                    {"name": "code_colis", "type" : ["null","string"]},
                                    {"name": "code_etat_lig_cde", "type" : ["null","string"]},
                                    {"name": "date_etat_lig_cde", "type" : ["null","long"]},
                                    {"name": "code_etab_log", "type" : ["null","string"]},
                                    {"name": "qte_cde", "type" : ["null","long"]},
                                    {"name": "qte_ano_stock", "type" : ["null","long"]},
                                    {"name": "qte_dem_prel", "type" : ["null","long"]},
                                    {"name": "qte_prel", "type" : ["null","long"]},
                                    {"name": "qte_expe_cli", "type" : ["null","long"]},
                                    {"name": "no_cde_cli", "type" : ["null","long"]},
                                    {"name": "flag_picking", "type" : ["null","string"]},
                                    {"name": "flag_envoi_site", "type" : ["null","string"]},
                                    {"name": "pud_pvttc_caisse", "type" : ["null","long"]},
                                    {"name": "valod_pud_pvttc_caisse", "type" : ["null","long"]},
                                    {"name": "code_operation", "type" : ["null","string"]},
                                    {"name": "taux_tva", "type" : ["null","double"]},
                                    {"name": "pud_remb", "type" : ["null","long"]},
                                    {"name": "valod_pud_remb", "type" : ["null","long"]},
                                    {"name": "code_pays_fab", "type" : ["null","string"]},
                                    {"name": "pud_remb_orig", "type" : ["null","long"]},
                                    {"name": "valod_pud_remb_orig", "type" : ["null","long"]},
                                    {"name": "pud_pvttc_caisse_orig", "type" : ["null","long"]},
                                    {"name": "valod_pud_pvttc_caisse_orig", "type" : ["null","long"]},
                                    {"name": "taux_tva_orig", "type" : ["null","double"]},
                                    {"name": "no_distro", "type" : ["null","string"]},
                                    {"name": "no_prio_expe", "type" : ["null","long"]},
                                    {"name": "qte_rec", "type" : ["null","long"]},
                                    {"name": "qte_ano_rec", "type" : ["null","long"]},
                                    {"name": "qte_ano_tri", "type" : ["null","long"]},
                                    {"name": "pud_pvttc_affichage_web", "type" : ["null","long"]},
                                    {"name": "valod_pud_pvttc_affichage_web", "type" : ["null","long"]},                                    
                                    {"name": "pud_pvttc_affichage_web_orig", "type" : ["null","long"]},
                                    {"name": "valod_pud_pvttc_affichage_web_orig", "type" : ["null","long"]},
                                    {"name": "qte_distro_expe", "type" : ["null","long"]},
                                    {"name": "type_tri", "type" : ["null","string"]},
                                    {"name": "code_etab_tri", "type" : ["null","string"]},
                                    {"name": "histo_details_commande",
                                        "doc": "on a 1 a N histo par detail",
                                        "type": {
                                            "type": "array",
                                            "items": {
                                                "type": "record",
                                                "name": "HistoDetailsCommande",
                                                "fields":[
                                                    {"name": "code_etat_lig_cde", "type" : "string"},
                                                    {"name": "date_etat_lig_cde", "type" : "long"},
                                                    {"name": "id_cmt", "type" : ["null","long"]}
                                                ]
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    }, 
                    {"name": "financement_commande",
                        "doc": "on a 1 a N financement par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "FinancementCommande",
                                "fields":[
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "type_financ", "type" : "string"},
                                    {"name": "mode_financ", "type" : "string"},
                                    {"name": "code_dev_financ", "type" : "string"},
                                    {"name": "no_ret", "type" : "long"},
                                    {"name": "mnt_financ", "type" : ["null","long"]},
                                    {"name": "valod_mnt_financ", "type" : ["null","long"]},
                                    {"name": "flag_envoi_site", "type" : ["null","string"]},
                                    {"name": "date_envoi_site", "type" : ["null","long"]},
                                    {"name": "flag_envoi_livcai", "type" : ["null","string"]},
                                    {"name": "date_envoi_livcai", "type" : ["null","long"]},
                                    {"name": "flag_val_site", "type" : ["null","string"]},
                                    {"name": "date_val", "type" : ["null","long"]},
                                    {"name": "mnt_pay_liv", "type" : ["null","long"]},
                                    {"name": "valod_mnt_pay_liv", "type" : ["null","long"]},
                                    {"name": "flag_financ_partiel", "type" : ["null","string"]},
                                    {"name": "flag_transaction_recu", "type" : ["null","string"]},
                                    {"name": "id_transaction", "type" : ["null","long"]},
                                    {"name": "flag_histo_livcai", "type" : ["null","string"]}
                                ]
                            }
                        }
                    },
                    {"name": "bon_commande",
                        "doc": "on a 1 a N bon utilise par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "BonCommande",
                                "fields":[
                                    {"name": "no_bon", "type" : "string"},
                                    {"name": "code_cli_site_web", "type" : ["null","string"]},
                                    {"name": "mnt_financ_bon", "type" : "long"},
                                    {"name": "valod_mnt_financ_bon", "type" : ["null","long"]},
                                    {"name": "code_dev_bon", "type" : "string"},
                                    {"name": "date_blocage_bon", "type" : ["null","long"]},
                                    {"name": "motif_blocage_bon", "type" : ["null","string"]}
                                ]
                            }
                        }
                    },
                    {"name": "bon_emis_commande",
                        "doc": "on a 1 a N bon emis par commande (suite retour)",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "BonEmisCommande",
                                "fields":[
                                    {"name": "no_bon", "type" : "string"},
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "no_ret", "type" : "long"},
                                    {"name": "val_bon", "type" : "long"},
                                    {"name": "valod_val_bon", "type" : ["null","long"]},
                                    {"name": "date_appli", "type" : "long"},
                                    {"name": "code_dev_bon", "type" : "string"},
                                    {"name": "code_cli_site_web", "type" : ["null","string"]}
                                ]
                            }
                        }
                    },
                    {"name": "expe_commande",
                        "doc": "on a 1 a N expedition par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "ExpeCommande",
                                "fields":[
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "no_reliquat", "type" : "string"},
                                    {"name": "code_etat_expe", "type" : "string"},
                                    {"name": "date_etat_expe", "type" : "long"},
                                    {"name": "no_cli", "type" : "long"},
                                    {"name": "no_adresse_liv", "type" : ["null","long"]},
                                    {"name": "code_type_liv", "type" : "string"},
                                    {"name": "flag_mono_cv", "type" : "string"},
                                    {"name": "flag_mono_entrepot", "type" : "string"},
                                    {"name": "no_pack", "type" : ["null","string"]},
                                    {"name": "code_relais", "type" : ["null","string"]},
                                    {"name": "code_pays", "type" : ["null","string"]},
                                    {"name": "code_transport", "type" : ["null","string"]},
                                    {"name": "mnt_frais_port", "type" : ["null","long"]},
                                    {"name": "valod_mnt_frais_port", "type" : ["null","long"]},
                                    {"name": "flag_envoi_ca", "type" : ["null","string"]},
                                    {"name": "flag_prel_cde", "type" : ["null","string"]},
                                    {"name": "flag_export", "type" : ["null","string"]},
                                    {"name": "code_transport_reel", "type" : ["null","string"]},
                                    {"name": "code_type_liv_reel", "type" : ["null","string"]}
                                ]
                            }
                        }
                    },
                    {"name": "retour_commande",
                        "doc": "on a 1 a N retour par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "RetourCommande",
                                "fields":[
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "no_reliquat", "type" : "string"},
                                    {"name": "no_ret", "type" : "long"},
                                    {"name": "date_saisie_ret", "type" : ["null","long"]},
                                    {"name": "flag_remb_bloque", "type" : ["null","string"]},
                                    {"name": "date_remb_deblocage", "type" : ["null","long"]},
                                    {"name": "code_transport_ret", "type" : ["null","string"]},
                                    {"name": "code_user_creation", "type" : ["null","string"]},
                                    {"name": "date_user_maj", "type" : ["null","long"]},
                                    {"name": "code_user_maj", "type" : ["null","string"]},
                                    {"name": "flag_envoi_ca", "type" : ["null","string"]},
                                    {"name": "date_envoi_ca", "type" : ["null","long"]},
                                    {"name": "cmt_ret", "type" : ["null","string"]},
                                    {"name": "flag_histo_ca", "type" : ["null","string"]},
                                    {"name": "details_retour_commande",
                                        "doc": "on a 1 a N detail pour chaque retour de commande",
                                        "type": {
                                            "type": "array",
                                            "items": {
                                                "type": "record",
                                                "name": "DetailsRetourCommande",
                                                "fields":[
                                                    {"name": "no_expe", "type" : "string"},
                                                    {"name": "no_reliquat", "type" : "string"},
                                                    {"name": "no_ligne", "type" : "long"},
                                                    {"name": "code_produit_reel", "type" : ["null","string"]},
                                                    {"name": "qte_ret", "type" : ["null","long"]},
                                                    {"name": "code_motif_ret", "type" : ["null","string"]},
                                                    {"name": "code_etab_emet", "type" : ["null","string"]},
                                                    {"name": "code_etab_dest", "type" : ["null","string"]},
                                                    {"name": "mnt_remb", "type" : ["null","long"]},
                                                    {"name": "valod_mnt_remb", "type" : ["null","long"]},
                                                    {"name": "code_motif_remb", "type" : ["null","string"]},
                                                    {"name": "code_mode_pay_remb", "type" : ["null","string"]},
                                                    {"name": "date_remb_dem", "type" : ["null","long"]},
                                                    {"name": "flag_remb_val", "type" : ["null","string"]},
                                                    {"name": "date_remb_val", "type" : ["null","long"]},
                                                    {"name": "flag_ces", "type" : ["null","string"]},
                                                    {"name": "flag_ces_val", "type" : ["null","string"]},
                                                    {"name": "date_extr_ces", "type" : ["null","long"]},
                                                    {"name": "no_colis_ces", "type" : ["null","long"]}
                                                ]
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    },
                    {"name": "colis",
                        "doc": "on a 1 a N colis par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "ColisCommande",
                                "fields":[
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "no_reliquat", "type" : "string"},
                                    {"name": "code_colis", "type" : "string"},
                                    {"name": "code_etat_colis", "type" : "string"},
                                    {"name": "date_etat_colis", "type" : ["null","long"]},
                                    {"name": "poids_colis", "type" : ["null","double"]},
                                    {"name": "code_transport", "type" : "string"},
                                    {"name": "code_colis_prec", "type" : ["null","string"]},
                                    {"name": "code_colis_suiv", "type" : ["null","string"]},
                                    {"name": "no_bord", "type" : ["null","string"]},
                                    {"name": "code_barre_colis", "type" : ["null","string"]},
                                    {"name": "date_envoi_edi", "type" : ["null","long"]},
                                    {"name": "no_ref_transaction_pay", "type" : ["null","string"]},
                                    {"name": "type_ret", "type" : ["null","string"]},
                                    {"name": "date_ret", "type" : ["null","long"]},
                                    {"name": "no_colis", "type" : ["null","long"]},
                                    {"name": "no_fact", "type" : ["null","string"]},
                                    {"name": "code_type_liv_reel", "type" : ["null","string"]},
                                    {"name": "mnt_pay_liv", "type" : ["null","long"]},
                                    {"name": "valod_mnt_pay_liv", "type" : ["null","long"]},
                                    {"name": "date_retrait_max", "type" : ["null","long"]},
                                    {"name": "nom_cli_retrait", "type" : ["null","string"]},
                                    {"name": "code_relais_reel", "type" : ["null","string"]},
                                    {"name": "code_colis_secondaire", "type" : ["null","string"]},
                                    {"name": "type_emballage", "type" : ["null","string"]},
                                    {"name": "financement_colis",
                                        "doc": "on a 1 a N financement par colis",
                                        "type": {
                                            "type": "array",
                                            "items": {
                                                "type": "record",
                                                "name": "FinancementColis",
                                                "fields":[
                                                    {"name": "type_financ", "type" : "string"},
                                                    {"name": "mode_financ", "type" : "string"},
                                                    {"name": "code_dev_financ", "type" : ["null","string"]},
                                                    {"name": "mnt_financ", "type" : ["null","double"]},
                                                    {"name": "valod_mnt_financ", "type" : ["null","double"]},
                                                    {"name": "trt_creation_enr", "type" : ["null","string"]},
                                                    {"name": "date_creation_enr", "type" : ["null","long"]},
                                                    {"name": "trt_maj_enr", "type" : ["null","string"]},
                                                    {"name": "date_maj_enr", "type" : ["null","long"]},
                                                    {"name": "flag_envoi_ca", "type" : ["null","string"]},
                                                    {"name": "date_envoi_ca", "type" : ["null","long"]},
                                                    {"name": "flag_transaction_recu", "type" : ["null","string"]},
                                                    {"name": "id_transaction", "type" : ["null","long"]},
                                                    {"name": "flag_histo_ca", "type" : ["null","string"]}
                                                ]
                                            }
                                        }
                                    },
                                    {"name": "histo_etat_colis",
                                        "doc": "on a 1 a N histo par colis",
                                        "type": {
                                            "type": "array",
                                            "items": {
                                                "type": "record",
                                                "name": "HistoEtatColis",
                                                "fields":[
                                                    {"name": "code_etat_colis", "type" : "string"},
                                                    {"name": "date_etat_colis", "type" : ["null","long"]},
                                                    {"name": "code_transport", "type" : "string"}
                                                ]
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    }
                ]
            }   
        }
    ]
}
``



write.avro fails on data frame

Error is
ravro:::write.avro(df, tf1)
Exception in thread "main" org.apache.avro.SchemaParseException: Enum has no symbols: {"name":"col_2","type":"enum","symbols":"d"}
at org.apache.avro.Schema.parse(Schema.java:1121)
at org.apache.avro.Schema.parse(Schema.java:1094)
at org.apache.avro.Schema$Parser.parse(Schema.java:927)
at org.apache.avro.Schema$Parser.parse(Schema.java:917)
at org.apache.avro.Schema.parse(Schema.java:966)
at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:91)
at org.apache.avro.tool.Main.run(Main.java:80)
at org.apache.avro.tool.Main.main(Main.java:69)

dump of df

df <-
structure(list(col_1 = 139.084976531123, col_2 = structure(1L, .Label = "d", class = "factor"),
col_3 = TRUE, col_4 = FALSE, col_5 = -11.3948273417181, col_6 = 90.2836501356233,
col_7 = structure(1L, .Label = "", class = "factor"), col_8 = structure(1L, .Label = "57be", class = "factor")), .Names = c("col_1",
"col_2", "col_3", "col_4", "col_5", "col_6", "col_7", "col_8"
), row.names = c(NA, -1L), class = "data.frame")

Another instance

Exception in thread "main" org.apache.avro.SchemaParseException: Enum has no symbols: {"name":"col_1","type":"enum","symbols":"_6f7a4bc347_ravro"}
at org.apache.avro.Schema.parse(Schema.java:1121)
at org.apache.avro.Schema.parse(Schema.java:1094)
at org.apache.avro.Schema$Parser.parse(Schema.java:927)
at org.apache.avro.Schema$Parser.parse(Schema.java:917)
at org.apache.avro.Schema.parse(Schema.java:966)
at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:91)
at org.apache.avro.tool.Main.run(Main.java:80)
at org.apache.avro.tool.Main.main(Main.java:69)

Dump

df <-
structure(list(col_1 = structure(1L, .Label = "6f7a4bc347", class = "factor"),
col_2 = structure(1L, .Label = "46f315f9", class = "factor"),
col_3 = -158.916518470489, col_4 = -72.4716823839384, col_5 = 34L,
col_6 = structure(1L, .Label = "6f7a", class = "factor"),
col_7 = -10L, col_8 = 10L), .Names = c("col_1", "col_2",
"col_3", "col_4", "col_5", "col_6", "col_7", "col_8"), row.names = c(NA,
-1L), class = "data.frame")

My theory from several example is failure occurs iff input is a data frame with a single row and at least one factor column

union of primitive and complex types

Greetings,

First - kind thanks for replying to the previous null issue.

In an attempt to avoid this null usage, we've rewritten the schema/data to leverage unions of primitive (string/int/etc) and complex (array/record) types. However, we're running into the following error:

Error in sch$type : $ operator is invalid for atomic vectors

This appears to happen during the sapply routine on the parsing of the union type. Below, JSON/schema pairs have been provided for an (1) int/array case and an (2) int/record case. Any insight or help would be wonderful.

Thanks in advance,
Sean

INT/ARRAY case

JSON:

{
  "dataset": 3,
  "app": {"array": ["423","jim","bob"]}
}

AVSC:

{
  "name": "employee_training_dataset",
  "type": "record",
  "fields": [
    {
      "name": "dataset",
      "type": "int"
    },
    {
      "name": "app",
      "type": ["int",
        {
          "type": "array",
          "items": "string"
        }
      ]
    }
  ]
}

INT/RECORD case

JSON:

{
  "dataset": 3,
  "app": {
    "app_record": {
      "id": 4,
      "address": "main"
    }
  }
}

AVSC:

{
  "name": "employee_training_dataset",
  "type": "record",
  "fields": [
    {
      "name": "dataset",
      "type": "int"
    },
    {
      "name": "app",
      "type": ["int",
        {
          "type": "record",
          "name": "app_record",
          "fields": [
            {
              "name": "id",
              "type": "int"
            },
            {
              "name": "address",
              "type": "string"
            }
          ]
        }
      ]
    }
  ]
}

Handling of NAs

Hi,
I have a data frame containing some NAs in one colum, When I write it out and read it back in, that column contains only NAs. What gives?

test case

df = 
structure(list(x = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1), y = structure(1:10, .Label = c("1", 
"2", "3", "4", "5", "6", "7", "8", "9", "10"), class = "factor"), 
    fac = structure(c(1L, NA, NA, 2L, 2L, 1L, NA, 2L, 1L, 1L), .Label = c("b", 
    "c"), class = "factor")), .Names = c("x", "y", "fac"), row.names = c(NA, 
-10L), class = "data.frame")

write.avro(df, "/tmp/testxx.avro")
read.avro("/tmp/testxx.avro")
   x  y  fac
1  1  1 <NA>
2  1  2 <NA>
3  1  3 <NA>
4  1  4 <NA>
5  1  5 <NA>
6  1  6 <NA>
7  1  7 <NA>
8  1  8 <NA>
9  1  9 <NA>
10 1 10 <NA>

Thanks

cannot install package in Rstudio

I am getting the following error RStudio for this
ERROR: dependencies 'bit64', 'rjson' are not available for package 'ravro'

  • removing 'C:/Users/xyz/Documents/R/win-library/4.0/ravro'
    Warning in install.packages : installation of package โ€˜C:/xyz/ravro/ravro-master/build/ravro_1.0.4.tar.gzโ€™ had non-zero exit status

Also when i run install_github("RevolutionAnalytics/ravro") it returns the error:
Error: Failed to install 'unknown package' from GitHub:
HTTP error 404.
Not Found

Did you spell the repo owner (RevolutionAnalytics) and repo name (ravro) correctly?

  • If spelling is correct, check that you have the required permissions to access the repo.

trouble on union with array/null case

Greetings!

When leveraging the union type for an array or null data, the read.avro routine fails. For reference, with the supplied test json and schema files (small_toy.zip), the avro java toolkit can successfully write an avro data file (fromjson), as well as write a json file from the created avro data file (tojson).

The issue appears to occur during the parse_avro.array call, which attempts to set schema_data. A debugger snapshot has been provided as well. It attempts to reference a nested array (schema) from a single atomic list (via "schema$").

This issue, if resolved, may also solve the issue found in a similar but larger test case (medium_toy.zip).

Thanks again,
Sean

medium_toy.zip
small_toy.zip
screen shot 2016-01-28 at 11 21 09 am

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.