Giter Site home page Giter Site logo

rippkgs's Introduction

Rippkgs

rippkgs is a search CLI utility for searching indexed nixpkgs expressions.

Usage

Installation

Once NixOS/nixpkgs#305125 is merged, if you have the commit you can use install using your preferred way of installing nixpkgs derivations. For example:

$ nix profile install nixpkgs#rippkgs nixpkgs#rippkgs-index
# alternatively, via the flake directly
$ nix profile install github:replit/rippkgs/v1.1.0

Alternatively, if you're using a flake to configure your system, you can add rippkgs via nixpkgs or as an input to your flake and add rippkgs to your environment packages. For example, in NixOS:

environment.systemPackages = [
  pkgs.rippkgs
  pkgs.rippkgs-index
  # alternatively, if via the flake input:
  inputs.rippkgs.packages.${system}.rippkgs
  inputs.rippkgs.packages.${system}.rippkgs-index
];

Generate an index

Generating an index may be done with the rippkgs-index cli:

rippkgs-index nixpkgs -o $XDG_DATA_HOME/rippkgs-index.sqlite

If you don't have a nixpkgs channel set (or would prefer to index a different channel), you'll have to explicitly pass the dir to the nixpkgs distribution:

rippkgs-index nixpkgs -o $XDG_DATA_HOME/rippkgs-index.sqlite ~/.nix-defexpr/channels/my-very-special-nixpkgs-channel

Alternatively, you can generate a registry using the flake output lib.genRegistry, which allows you to avoid recursive-nix problems:

$ nix eval -L .#lib.genRegistry --apply 'f: f (import <nixpkgs> { })' --impure --json >registry.json
$ rippkgs-index registry -o rippkgs-index.sqlite registry.json

Searching

Use the rippkgs cli to search for appropriate packages:

rippkgs rustc

Comparison

nix-env -q is historically the command that's used to achieve what rippkgs achieves, but the nix evaluation cost is high. When in an environment where reactiveness is desirable, it's better to pay the initial cost of generating the index (see results below).

nix-index is similar in that it operates on a generated database, but requires derivations to be built in order to generate the index, and not all package information is stored in the index. This means you have to augment nix-locate results with nix-env in order to get additional information about a package, like its version or the nixpkgs registry description for the package.

https://search.nixos.org is the closest approximate tool to rippkgs, but unfortunately it comes with a few shortcomings:

  1. The index only tracks up-to-date nixpkgs distributions, which means you must resort to nix-env to get results accurate to the nixpkgs release you're using.
  2. Search is only available as a web service with only HTML responses to its HTTP api, which means you have to parse the HTML response to your programmatic requests.
  3. It uses Elastic Search, which is fine for a widely-used service but doesn't work well when providing highly-localized results.

Performance

Using nixpkgs@b550fe4b4776908ac2a861124307045f8e717c8e on an aarch64-darwin with 16gb of memory:

$ time ./target/release/rippkgs-index nixpkgs -o nixpkgs.sqlite
evaluated registry in 289.8638 seconds
parsed registry in 0.0761 seconds
wrote index in 0.1432 seconds
./target/release/rippkgs-index nixpkgs -o nixpkgs.sqlite  108.35s user 45.50s system 53% cpu 4:50.20 total

$ time ./target/release/rippkgs -i nixpkgs.sqlite rustc 1>/dev/null
got results in 44 ms
./target/release/rippkgs -i nixpkgs.sqlite rustc > /dev/null  0.03s user 0.02s system 89% cpu 0.051 total

rippkgs's People

Contributors

cdmistman avatar

Stargazers

 avatar  avatar Aurora Dawn avatar  avatar TheMattaBase avatar Dich0tomy avatar MiSumiSumi avatar  avatar Garrett George avatar shard avatar Mika Cohen avatar Yves M. Lujan avatar  avatar Suraj avatar dusk avatar Qyriad avatar Christian Friedow avatar Zacchary Dempsey-Plante avatar RGBCube avatar Florian Nagel avatar Bob Patterson Jr avatar Mat avatar  avatar ngn999 avatar aemogie. avatar Jake Chvatal avatar Mihai Fufezan avatar Ali Can YILDIZ avatar Florian Peter avatar toyboot4e avatar João Figueira avatar Guilhem Fauré avatar Yash Garg avatar Alex Moiseenko avatar  avatar  avatar Evan Stoll avatar Sandalots avatar Jan-Erik Rediger avatar David Hagege avatar Niklas Korz avatar Ankesh Bharti avatar Stefanie  avatar Will Bush avatar Gabriel Volpe avatar  avatar Alan Liu avatar Elia Berrettini avatar omar avatar  avatar Ilmari Vacklin avatar Francis Begyn avatar Josh Kingsley avatar Martin Myrseth avatar Nikita avatar Evan Richter avatar raffaele messuti avatar Miguel Madrid-Mencía avatar Christina Sæøåræånsæn avatar Shaun Newman avatar Juri Hahn avatar José Ribeiro avatar Ivan Tham avatar Ajam avatar Ruslan Prakapchuk avatar Tristan avatar Uwe Sommerlatt avatar thomas avatar Rafael Ristovski avatar Nikita Wootten avatar Nate Smith avatar t-chab avatar Jordan Arentsen avatar Nimrod avatar Brad Svercl avatar Shiryel avatar Matthias Meschede avatar Dmitrii Manik avatar July avatar le Duc Banal avatar George Kontridze avatar Kamilla 'ova avatar John Olheiser avatar Mathis Wellmann avatar Makar Kungurov avatar Andrey Kuznetsov avatar Justus K avatar Viacheslav Vasilyev avatar Romain Gautier avatar Tsiry Sandratraina avatar Justin Restivo avatar Vlad M. avatar  avatar Tom avatar j-k avatar Milos Gajdos avatar Connor Brewster avatar James avatar h7x4 avatar Jappie3 avatar

Watchers

 avatar Ertan Dogrultan avatar David Hoang avatar Alexis Cervantes avatar Matt Legrand avatar Faris Masad avatar James Austin avatar Harrison Unruh avatar  avatar Prahathess avatar Arnav Bansal avatar Connor Brewster avatar Olivia Chen avatar Edgar Hernandez avatar  avatar  avatar  avatar

Forkers

ryantm jeffwkm

rippkgs's Issues

Add to nixpkgs?

Hello,

Can this be added to nixpkgs soon so we can get the builds cached?

Thanks!

Feature request: `command-not-found` implementation

First of all, thank you @replit for creating rippkgs and releasing it as an open-source project :)

In your release blog post, it is mentioned that "you’ve incorporated rippkgs into your command-not-found hook".

Given the scope of rippkgs (and the comparison to nix-index), would it be possible to provide a minimal command-not-found handler implementation that makes use of rippkgs (assuming Replit's implementation is too specific to be shared)?

This would be very useful to users of Nixpkgs/devShells in general, but particularly to NixOS users in flake-only (i.e. no channels) configurations [1], who usually resort to nix-index or a custom programs.command-not-found.dbPath [2].

Once again, thank you for rippkgs!

package derivations missing `meta.mainProgram` attribute, causing deprecation warnings

When I use lib.getExe from Nixpkgs on either the rippkgs or rippkgs-index packages, I see the following warnings:

trace: warning: getExe: Package "rippkgs-1.1.0" does not have the meta.mainProgram attribute. We'll assume that the main program has the same name for now, but this behavior is deprecated, because it leads to surprising errors when the assumption does not hold. If the package has a main program, please set `meta.mainProgram` in its definition to make this warning go away. Otherwise, if the package does not have a main program, or if you don't control its definition, use getExe' to specify the name to the program, such as lib.getExe' foo "bar".
trace: warning: getExe: Package "rippkgs-index-1.1.0" does not have the meta.mainProgram attribute. We'll assume that the main program has the same name for now, but this behavior is deprecated, because it leads to surprising errors when the assumption does not hold. If the package has a main program, please set `meta.mainProgram` in its definition to make this warning go away. Otherwise, if the package does not have a main program, or if you don't control its definition, use getExe' to specify the name to the program, such as lib.getExe' foo "bar".

Example

This is especially noisy when using lib.getExe on either package in a custom script to be used in a devshell. It's not really clear from the snippet above, but the wrapped output of this paragraph-long warning text can span several lines.

Note that rippkgs.packages here is equivalent to flake-parts' inputs'.rippkgs.packages where inputs' is the argument in perSystem.

(pkgs.writeShellScriptBin "rippkgs-index-flake" ''
  # TEST: ${lib.getExe rippkgs.packages.rippkgs}
  ${lib.getExe rippkgs.packages.rippkgs-index} nixpkgs \
    -o $XDG_DATA_HOME/rippkgs-index.sqlite \
    ${inputs.nixpkgs.outPath}
'')

Add aarch64-linux target.

nix run github:replit/rippkgs#rippkgs
error: flake 'github:replit/rippkgs' does not provide attribute 'apps.aarch64-linux.rippkgs', 'packages.aarch64-linux.rippkgs', 'legacyPackages.aarch64-linux.rippkgs' or 'rippkgs

`nix eval` failed while indexing

❯ rippkgs-index nixpkgs                                              
evaluated registry in 309.7863 seconds
thread 'main' panicked at src/bin/index/main.rs:198:9:
`nix eval` failed: 
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Let me know how I can get more info to help you debug.

`rippkgs-index` provides no output feedback to user during indexing

When I run rippkgs-index as instructed in the README, I expect to see some kind of confirmation that anything at all is happening.

Currently, the program runs but appears to hang indefinitely because it echoes no output messages.

Until I ran htop and saw high resource usage from the Nix daemon and from the nix eval process, I wasn't sure whether rippkgs-index was working or whether I made some mistake.

Example

Note that I have my NixOS configuration's flake inputs symlinked to /etc/nix/inputs/. I do not think that is related to the issue, however, because I also encountered the same behavior when using inputs.nixpkgs.outPath in a custom script.

rippkgs-index nixpkgs -o $XDG_DATA_HOME/rippkgs-index.sqlite /etc/nix/inputs/nixpkgs

indexing process causes unresponsive system until OOM-killed

As I was running rippkgs-index for the first time, my desktop session froze / became unresponsive. Eventually, the nix eval process was terminated by the OOM killer.

I then saw that my terminal application no longer had any active sessions/frames. Unfortunately, the index output file is nowhere to be found.

I've included some logs below. Let me know if you need more info. However, I feel reluctant to make another attempt to build an index until this issue is fixed or there is some other way to provide an index (#30).

Apr 07 23:27:27 tuvok kernel: nix-daemon invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Apr 07 23:27:27 tuvok kernel: CPU: 7 PID: 418137 Comm: nix-daemon Tainted: G S                 6.6.0-asahi #1-NixOS
Apr 07 23:27:27 tuvok kernel: Hardware name: Apple MacBook Air (13-inch, M2, 2022) (DT)
Apr 07 23:27:27 tuvok kernel: Call trace:
Apr 07 23:27:27 tuvok kernel:  dump_backtrace+0x94/0x114
Apr 07 23:27:27 tuvok kernel:  show_stack+0x18/0x24
Apr 07 23:27:27 tuvok kernel:  dump_stack_lvl+0x48/0x60
Apr 07 23:27:27 tuvok kernel:  dump_stack+0x18/0x24
Apr 07 23:27:27 tuvok kernel:  dump_header+0x44/0x1bc
Apr 07 23:27:27 tuvok kernel:  oom_kill_process+0x130/0x2d0
Apr 07 23:27:27 tuvok kernel:  out_of_memory+0xd8/0x344
Apr 07 23:27:27 tuvok kernel:  __alloc_pages_may_oom+0x11c/0x1b4
Apr 07 23:27:27 tuvok kernel:  __alloc_pages_slowpath.constprop.0+0x714/0x924
Apr 07 23:27:27 tuvok kernel:  __alloc_pages+0x214/0x29c
Apr 07 23:27:27 tuvok kernel:  __folio_alloc+0x14/0x20
Apr 07 23:27:27 tuvok kernel:  __filemap_get_folio+0xe4/0x300
Apr 07 23:27:27 tuvok kernel:  filemap_fault+0x110/0x7b0
Apr 07 23:27:27 tuvok kernel:  __do_fault+0x3c/0x12c
Apr 07 23:27:27 tuvok kernel:  do_read_fault+0x50/0x240
Apr 07 23:27:27 tuvok kernel:  do_pte_missing+0x168/0x278
Apr 07 23:27:27 tuvok kernel:  __handle_mm_fault+0x214/0x27c
Apr 07 23:27:27 tuvok kernel:  handle_mm_fault+0x68/0x27c
Apr 07 23:27:27 tuvok kernel:  do_page_fault+0x204/0x4c0
Apr 07 23:27:27 tuvok kernel:  do_translation_fault+0x38/0x4c
Apr 07 23:27:27 tuvok kernel:  do_mem_abort+0x44/0x94
Apr 07 23:27:27 tuvok kernel:  el0_ia+0x5c/0x170
Apr 07 23:27:27 tuvok kernel:  el0t_64_sync_handler+0xcc/0x12c
Apr 07 23:27:27 tuvok kernel:  el0t_64_sync+0x190/0x194
Apr 07 23:27:27 tuvok kernel: Mem-Info:
Apr 07 23:27:27 tuvok kernel: active_anon:640 inactive_anon:945204 isolated_anon:0
                               active_file:170 inactive_file:11 isolated_file:0
                               unevictable:28 dirty:0 writeback:0
                               slab_reclaimable:7580 slab_unreclaimable:7817
                               mapped:11575 shmem:13491 pagetables:4149
                               sec_pagetables:0 bounce:0
                               kernel_misc_reclaimable:0
                               free:7396 free_pcp:428 free_cma:0
Apr 07 23:27:27 tuvok kernel: Node 0 active_anon:10240kB inactive_anon:15123264kB active_file:2720kB inactive_file:176kB unevictable:448kB isolated(anon):0kB isolated(file):0kB mapped:185200kB dirty:0kB writeback:0kB shmem:215856kB writeback_tmp:0kB kernel_stack:22096kB pagetables:66384kB sec_pagetables:0kB all_unreclaimable? no
Apr 07 23:27:27 tuvok kernel: DMA free:118336kB boost:70960kB min:86816kB low:102544kB high:118272kB reserved_highatomic:32768KB active_anon:10240kB inactive_anon:15123264kB active_file:0kB inactive_file:0kB unevictable:448kB writepending:0kB present:16049696kB managed:15745344kB mlocked:416kB bounce:0kB free_pcp:6848kB local_pcp:912kB free_cma:0kB
Apr 07 23:27:27 tuvok kernel: lowmem_reserve[]: 0 0 0 0
Apr 07 23:27:27 tuvok kernel: DMA: 87*16kB (EH) 24*32kB (EH) 1039*64kB (UEH) 313*128kB (UE) 26*256kB (UE) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 115376kB
Apr 07 23:27:27 tuvok kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 07 23:27:27 tuvok kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB
Apr 07 23:27:27 tuvok kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 07 23:27:27 tuvok kernel: 13796 total pagecache pages
Apr 07 23:27:27 tuvok kernel: 0 pages in swap cache
Apr 07 23:27:27 tuvok kernel: Free swap  = 0kB
Apr 07 23:27:27 tuvok kernel: Total swap = 0kB
Apr 07 23:27:27 tuvok kernel: 1003106 pages RAM
Apr 07 23:27:27 tuvok kernel: 0 pages HighMem/MovableOnly
Apr 07 23:27:27 tuvok kernel: 19022 pages reserved
Apr 07 23:27:27 tuvok kernel: 4096 pages cma reserved
Apr 07 23:27:27 tuvok kernel: Tasks state (memory values in pages):
[...output elided...]
Apr 07 23:27:27 tuvok kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-org.codeberg.dnkl.foot-13409.scope,task=nix,pid=418134,uid=1000
Apr 07 23:27:27 tuvok kernel: Out of memory: Killed process 418134 (nix) total-vm:10673712kB, anon-rss:10536784kB, file-rss:512kB, shmem-rss:0kB, UID:1000 pgtables:5296kB oom_score_adj:100
Apr 07 23:27:27 tuvok gnome-shell[2185]: libinput error: event4  - kanata: client bug: event processing lagging behind by 39ms, your system is too slow
Apr 07 23:27:27 tuvok gnome-shell[2185]: libinput error: event4  - kanata: client bug: event processing lagging behind by 250ms, your system is too slow
Apr 07 23:27:27 tuvok gnome-shell[2185]: libinput error: client bug: timer event2 hold: scheduled expiry is in the past (-56ms), your system is too slow
Apr 07 23:27:27 tuvok systemd[2070]: app-gnome-org.codeberg.dnkl.foot-13409.scope: A process of this unit has been killed by the OOM killer.
Apr 07 23:27:27 tuvok systemd[1]: [email protected]: A process of this unit has been killed by the OOM killer.
Apr 07 23:27:27 tuvok systemd-udevd[762]: port0: Worker [828121] processing SEQNUM=4481 is taking a long time
Apr 07 23:27:27 tuvok nix-daemon[418137]: unexpected Nix daemon error: error: interrupted by the user

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.