apauley/hledger-flow

An hledger/ledger-cli workflow focusing on automated statement import and classification

Hledger Flow

TravisCI	CircleCI

What is it?

hledger-flow is a command-line program that gives you a guided Hledger workflow. It is important to note that most of the heavy lifting is done by the upstream hledger project. For example, hledger-flow cares about where you put your files for long-term maintainability, but the actual conversion to classified accounting journals is done by hledger.

hledger-flow focuses on automated processing of electronic statements as much as possible, as opposed to manually adding your own hledger journal entries. Manual entries are still possible, it just saves time in the long run to automatically process a statement whenever one is available.

Within hledger-flow you will keep your original bank statements around permanently as input, and generate classified Hledger journals each time you run the program. The classification is done with hledger’s rules files, and/or your own script hooks.

Keeping the original statements means that you never have to worry too much about “am I doing this accounting thing right?” or “what happens if I make a mistake?”. If you want to change your mind about some classification, or if you made a mistake, you just change your classification rules, and run the program again.

It started when I realized that the scripts I wrote while playing around with adept’s Full-fledged Hledger aren’t really specific to my own finances, and can be shared.

Overview of the Basic Workflow

1. Save an input transaction file (typically CSV) to a specific directory.
2. Add an hledger rules file. Include some classification rules if you want.
3. Run hledger-flow import

Add all your files to your favourite version control system.

The generated journal that you most likely want to use as your LEDGER_FILE is called all-years.journal. This has include directives to all the automatically imported journals, as well as includes for your own manually managed journal entries.

In a typical software project we don’t add generated files to version control, but in this case I think it is a good idea to add all the generated files to version control as well - when you inevitably change something, e.g. how you classify transactions in your rules file, then you can easily see if your change had the desired effect by looking at a diff.

Who should use this?

hledger-flow is intended for you if:

• You are interested in getting started with hledger or ledger-cli and you wouldn’t mind pointers to the right docs along the way.
• You want a way to organise your finances into a structure that will be maintainable over the long term.
• You want to automate as much as possible when dealing with your financial life.
• You don’t mind writing some scripts when needed, as long as it saves you time over the long term.
• You want the ability to model your entire financial life in one tool, as opposed to just the parts that some online tool currently supports.
• You appreciate the fact that all your financial information stays within your control.

About This Documentation

The rest of this file documents how to use hledger-flow, and it has probably outgrown what should be in a README file.

If you can spare some time to contribute to this project, please consider converting these docs to something more suitable, such as Read the Docs.

How do I install it?

The easiest way to get it running is to download the latest release for your OS (Linux or Mac OS X), and copy the hledger-flow executable to a directory in your PATH. Then just run it and see what it tells you to do.

You can also compile it yourself by following the build instructions.

Windows Support

Currently hledger-flow does not work on Windows.

This list of issues describes some of the details of what doesn’t work.

I believe it wouldn’t take too much effort to fix those issues, but I’m going to leave Windows support for other contributors.

Please send me some pull requests if you would like hledger-flow to work on Windows.

How Stable is it?

We’re not close to a 1.0 release yet, which means that we can still make changes if needed.

As an example, the command-line switches we use will probably change over time. Some switches change the behaviour of the program - the default behaviour will probably change between releases. The names of these command-line options can change, or they can be removed when it is no longer needed.

That being said, some parts have been used and tested extensively and are likely to remain stable. Have a look at the “Stability of this Feature” sections in the feature reference below.

I add future work, ideas and thoughts as Github issues and in TODO.org, so have a look there for more clues as to what may likely change.

Let me know if you can think of some improvements.

Getting Started

Have a look at the detailed step-by-step instructions.

You can see the example imported financial transactions as it was generated by the step-by-step instructions here:

https://github.com/apauley/hledger-flow-example

Feature Reference

Input Files

Your input files will probably be CSV files with a line for each transaction, although other file types will work fine if you use a preprocess or a construct script that can read them. These scripts are explained later.

We mostly use conventions based on a predefined directory structure for your input statements.

For example, assuming you have a savings account at mybank, you’ll put your first CSV statement here: import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv.

Some people may want to include accounts belonging to their spouse as part of the household finances: import/spouse/otherbank/checking/1-in/2018/987654321_2018-06-30.csv.

More About Input Files

All files and directories under the import directory are related to the automatic importing and classification of transactions.

The directory directly under import is meant to indicate the owner or custodian of the accounts below it. It mostly has an impact on reporting. You may want to have separate reports for import/mycompany and import/personal.

Below the directory for the owner we can indicate where an account is held. For a bank account you may choose to name it import/john/mybank.

If your underground bunker filled with gold has CSV statements linked to it, then you can absolutely create import/john/secret-treasure-room.

Under the directory for the financial institution, you’ll have a directory for each account at that institution, e.g. import/mycompany/bigbankinc/customer-deposits and import/mycompany/bigbankinc/expense-account.

Next you’ll create a directory named 1-in. This is to distinguish it from 2-preprocessed and 3-journal which will be auto-generated later.

Under 1-in you’ll create a directory for the year, e.g. 2018, and within that you can copy the statements for that year: import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv

Stability of this Feature

The basic owner/bank/account/year structure has been used and tested fairly extensively, I don’t expect a need for it to change.

I’m open to suggestions for improvement though.

Rules Files

If your input file is in CSV format, or converted to CSV by your preprocess script, then you’ll need an hledger rules file.

hledger-flow will try to find a rules file for each statement in a few places. The same rules file is typically used for all statements of a specific account, or even for all accounts of the same specific bank.

• A global rules file for any mybank statement can be saved here: import/mybank.rules
• A rules file for all statements of a specific account: import/spouse/bigbankinc/savings/bigbankinc-savings.rules

Statement-specific Rules Files

What happens if some of the statements for an account has a different format than the others?

This can happen if you normally get your statements directly from your bank, but some statements you had to download from somewhere else, like Mint, because your bank is being daft with older statements.

In order to tell hledger-flow that you want to override the rules file for a specific statement, you need to add a suffix, separated by an underscore (_) and starting with the letters rfo (rules file override) to the filename of that statement.

For example: assuming you’ve named your statement 99966633_20171223_1844_rfo-mint.csv.

hledger-flow will look for a rules file named rfo-mint.rules in the following places:

• in the import directory, e.g. import/rfo-mint.rules
• in the bank directory, e.g. import/john/mybank/rfo-mint.rules
• in the account directory, e.g. import/john/mybank/savings/rfo-mint.rules

Example rules file usage

A common scenario is multiple accounts that share the same file format, but have different account1 directives.

One possible approach would be to include a shared rules file in your account-specific rules file.

If you are lucky enough that all statements at mybank share a common format across all accounts, then you can include a rules file that just defines the parts that are shared across accounts.

Two accounts at mybank may have rules files similar to these.

A checking account at mybank:

# Saved as: import/john/mybank/checking/mybank-checking.rules
include ../../../mybank-shared.rules
account1 Assets:Current:John:MyBank:Checking

Another account at mybank:

# Saved as: import/alice/mybank/savings/mybank-savings.rules
include ../../../mybank-shared.rules
account1 Assets:Current:Alice:MyBank:Savings

Where import/mybank-shared.rules may define some shared attributes:

skip 1

fields date, description, amount, balance

date-format %Y-%m-%d
currency $

Another possible approach could be to use your preprocess script to write out a CSV file that has extra fields for account1 and account2.

You could then create the above mentioned global import/mybank.rules with the fields defined more or less like this:

fields date, description, amount, balance, account1, account2

Stability of this Feature

Rules files are a stable feature within hledger, and we’re just using the normal hledger rules files. The account, bank and statement-specific rules files have been used and tested fairly extensively, I don’t expect this to change.

Let me know if you think it should change.

Opening and Closing Balances

Opening Balances

hledger-flow looks for a file named YEAR-opening.journal in each account directory, where YEAR corresponds to an actual year directory, eg. 1983 (if you have electronic statements dating back to 1983). Example: import/john/mybank/savings/1983-opening.journal

If it exists the file will automatically be included at the beginning of the generated journal include file for that year.

You need to edit this file for each account to specify the opening balance at the date of the first available transaction.

An opening balance may look something like this:

2018-06-01 Savings Account Opening Balance
    assets:Current:MyBank:Savings               $102.01
    equity:Opening Balances:MyBank:Savings

A Note of Caution Regarding Closing Balances

When closing your balances it may result in some hledger queries showing zero-values, or there could be issues with balance assertions.

Please have a look at the upstream hledger documentation on closing balances, e.g here: https://hledger.org/hledger.html#close-usage

Some of the gotchas you may run into are also described in this hledger-flow issue.

Closing Balances

Similar to opening balances, hledger-flow looks for an optional file named YEAR-closing.journal in each account directory. Example: import/john/mybank/savings/1983-closing.journal

If it exists the file will automatically be included at the end of the generated journal include file for that year.

A closing balance may look something like this:

2018-06-01 Savings Account Closing Balance
    assets:Current:MyBank:Savings               $-234.56 = $0.00
    equity:Closing Balances:MyBank:Savings

Example Opening and Closing Journal Files

As an example, assuming that the relevant year is 2019 and hledger-flow is about to generate import/john/mybank/savings/2019-include.journal, then one or both of the following files will be added to the include file if they exist:

1. import/john/mybank/savings/2019-opening.journal
2. import/john/mybank/savings/2019-closing.journal

The opening.journal will be included just before the other included entries, while the closing.journal will be included just after the other entries in that include file.

An include file may look like this:

cat import/john/mybank/savings/2019-include.journal

### Generated by hledger-flow - DO NOT EDIT ###

include 2019-opening.journal
include 3-journal/2019/123456789_2019-01-30
include 2019-closing.journal

Stability of this Feature

Closing balances sometimes result in unexpected query results. In future we may change how/where the generated files include the closing journal.

We may also need to suggest some naming conventions for opening and closing balances so that reports can exclude some of these transactions.

It is also possible that we might want to change the name/location of the closing journal, but we’ll try to avoid this if possible, because that would require users to rename their existing files.

Price Files

hledger-flow looks for price files to include in each yearly include file.

For example, the presence of a file named ${BASE}/prices/2020/prices.journal will result in some extra include file magic.

The rest of this section assumes you’ll have a file named prices/2020/prices.journal which contains price data for the year 2020. The prices directory should be right at the top of your hledger-flow base directory, next to the import directory.

hledger-flow does not care how the price files got there, it only cares that you should have a separate file per year, and that it follows the above naming convention.

Here is an example script which downloads prices and follows the naming convention: https://gist.github.com/apauley/398fa031c202733959af76b3b8ce8197

After running an import with available price files you’ll see a line has been added to import/2020-include.journal:

include ../prices/2020/prices.journal

Hledger Directives

Hledger allows you to specify some useful directives which affect things such as number formatting.

A convenient place to put these directives within hledger-flow is a file named directives.journal (in your hledger-flow base directory).

If it exists hledger-flow will include it within the all-years.journal:

cat all-years.journal

### Generated by hledger-flow - DO NOT EDIT ###

include directives.journal
include import/all-years.journal

The preprocess Script

Sometimes the statements you get from your bank is less than suitable for automatic processing. Or maybe you just want to make it easier for the hledger rules file to do its thing by adding some useful columns.

If you put a script called preprocess in the account directory, e.g. import/john/mybank/savings/preprocess, then hledger-flow will call that script for each input statement.

The preprocess script will be called with 4 positional parameters:

1. The path to the input statement, e.g. import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv
2. The path to an output file that can be sent to hledger, e.g. import/john/mybank/savings/2-preprocessed/2018/123456789_2018-06-30.csv
3. The name of the bank, e.g. mybank
4. The name of the account, e.g. savings
5. The name of the owner, e.g. john

Your preprocess script is expected to:

• read the input file
• write a new output file at the supplied path that works with your rules file
• be idempotent. Running preprocess multiple times on the same files will produce the same result.

Stability of this Feature

Stable and tested.

The construct Script

If you need even more power and flexibility than what you can get from the preprocess script and hledger’s CSV import functionality, then you can create your own custom script to construct transactions exactly as you need them.

At the expense of more construction work for you, of course.

The construct script can be used in addition to the preprocess script, or on it’s own. But since the construct script is more powerful than the preprocess script, you could tell your construct script to do anything that the preprocess script would have done.

Save your construct script in the account directory, e.g. import/john/mybank/savings/construct.

hledger-flow will call your construct script with 5 positional parameters:

1. The path to the input statement, e.g. import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv
2. A “-” (indicating that output should be sent to stdout)
3. The name of the bank, e.g. mybank
4. The name of the account, e.g. savings
5. The name of the owner, e.g. john

Your construct script is expected to:

• read the input file
• generate your own hledger journal transactions
• be idempotent. Running construct multiple times on the same files should produce the same result.
• send all journals to stdout. hledger-flow will pipe your standard output into hledger which will format it and save it to an output file.

You can still use stderr in your construct script for any other output that you may want to see.

Stability of this Feature

Stable and tested.

Manually Managed Journals

Not every transaction in your life comes with CSV statements.

Sometimes you just need to add a transaction for that time you loaned a friend some money.

hledger-flow looks for pre-import and post-import files related to each generated include file as part of the import.

You can enter your own transactions manually into these files.

You can run hledger-flow import --verbose to see exactly which files are being looked for.

As an example, assuming that the relevant year is 2019 and hledger-flow is about to generate import/john/2019-include.journal, then one or both of the following files will be added to the include file if they exist:

1. import/john/_manual_/2019/pre-import.journal
2. import/john/_manual_/2019/post-import.journal

The pre-import.journal will be included just before the other included entries, while the post-import.journal will be included just after the other entries in that include file.

An include file may look like this:

cat import/john/2019-include.journal

### Generated by hledger-flow - DO NOT EDIT ###

include _manual_/2019/pre-import.journal
include mybank/2019-include.journal
include otherbank/2019-include.journal
include _manual_/2019/post-import.journal

Stability of this Feature

It works, but the naming of _manual_ looks a bit weird. Should it be changed?

Validating an hledger repository using Github Actions

The following example was contributed by Amitai Burstein:

# .github/workflows/hledger-flow.yml

name: Validate hledger-flow

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1

    - name: Install hledger
      run: docker pull dastapov/hledger

    - name: Install hledger-flow
      run: curl -L https://github.com/apauley/hledger-flow/releases/download/v0.12.4.0/hledger-flow_Linux_x86_64_v0.12.4.0_4b9b027.tar.gz | tar xvz && mv hledger-flow_Linux_x86_64_v0.12.4.0_4b9b027/hledger-flow .

    - name: Grant permissions to create files
      run: chmod 777 -R ./my-finances

    - name: Test hledger file
      run: docker run --name="ledger" -v $(pwd):/data dastapov/hledger ./hledger-flow import ./my-finances

Compatibility with Ledger

When writing out the journal include files, hledger-flow sorts the include statements by filename.

Ledger fails any balance assertions when the transactions aren’t included in chronological order.

An easy way around this is to name your input files so that March’s statement is listed before December’s statement.

Another option is to add --permissive to any ledger command.

So you should easily be able to use both ledger and hledger on these journals if you take care to avoid the few incompatibilities which exists (eg in your rules files or manual journals).

Project Goals

My hledger files started to collect a bunch of supporting code that weren’t really specific to my financial situation.

I want to extract and share as much as possible of that supporting code.

Adept’s goals also resonated with me:

• Tracking expenses should take as little time, effort and manual work as possible
• Eventual consistency should be achievable: even if I can’t record something precisely right now, maybe I would be able to do it later, so I should be able to leave things half-done and pick them up later
• Ability to refactor is a must. I want to be able to go back and change the way I am doing things, with as little effort as possible and without fear of irrevocably breaking things.

I’ve given a talk at Lambda Luminaries Johannesburg featuring hledger and hledger-flow.

Contributing to Hledger Flow

Have a look at the contribution guidelines.

FAQ

How do you balance transfers between 2 accounts when you have statements for both accounts?

The Problem

In your primary bank account you’ve happily been classifying transfers to a secondary account as just Expenses:OtherAccount.

But you’ve recently started processing the statements from the second account as well so that you can classify those expenses more accurately.

And now the balances of these two accounts are all wrong when the statements of each account deals with money transferred between these two accounts.

In bank1.journal, imported from bank1.csv:

2018/11/09 Transfer from primary account to secondary account
    Assets:Bank1:Primary    $-200
    Assets:Bank2:Secondary

In bank2.journal, imported from bank2.csv:

2018/11/09 Transfer from primary account to secondary account
    Assets:Bank2:Secondary  $200
    Assets:Bank1:Primary

The Solution

As soon as you start importing statements for both accounts you will have to introduce an intermediate account for classification between these two accounts.

I use Assets:Transfers:*.

And we may have reports looking at these transfers accounts at some point, you should consider using the same names.

The above example then becomes as follows.

In bank1.journal, imported from bank1.csv:

2019-05-18 Transfer from primary account to secondary account
    Assets:Bank1:Primary         $-200
    Assets:Transfers:Bank1Bank2

In bank2.journal, imported from bank2.csv:

2019-05-18 Transfer from primary account to secondary account
    Assets:Bank2:Secondary       $200
    Assets:Transfers:Bank1Bank2

Any posting to Assets:Transfers:* indicates an in “in-flight” amount. You would expect the balance of Assets:Transfers to be zero most of the time. Whenever it isn’t zero it means that you either don’t yet have the other side of the transfer, or that something is wrong in your rules.

You could theoretically just use Assets:Transfers without any subaccounts, but I found it useful to use subaccounts. Because then the subaccounts can show me where I should look for any missing transfer transaction.

I typically use sorted names as the subaccount (Python code sample):

"Assets:Transfers:" + "".join(sorted(["Bank2", "Bank1"]))

External references

This approach is based on what is described in Full-fledged hledger: https://github.com/adept/full-fledged-hledger/wiki/Adding-more-accounts#lets-make-sure-that-transfers-are-not-double-counted

The question was first asked in issue #51.

How does hledger-flow differ from Full-fledged Hledger?

Full-fledged Hledger is a brilliant system, and hledger-flow continues to learn much from it.

It has great documentation that does an excellent job of not only showing how things can be done, but also why it is such a great idea.

hledger-flow can be seen as a specific implementation of the Full-fledged Hledger system, with a few implementation details that are different.

Full-fledged Hledger	Hledger Flow
FFH is a tutorial with helper scripts that you can start using and adapt to your needs.	I started with FFH, and changed bits and pieces over time to suit my needs. The “owner/bank/account” structure for example.
FFH is more open-ended: you can start with the basic scripts and over time turn it into something that solves your needs exactly. But you’ll also end up with more code that you need to maintain yourself.	Hledger Flow is more opinionated and less open-ended. For example, you have to adopt the “owner/bank/account” structure precisely as specified. But this allows Hledger Flow to do more work for you.
FFH uses scripts and Haskell/Shake build files that you can easily modify as you go along, but this requires a Haskell runtime to be installed everywhere it needs to run. The included docker image helps to make it less of an issue.	Hledger Flow distributes a compiled binary. This means users or deployment targets don’t need extra dependencies installed, they can just run a CLI program. This also provides a clearer distinction between what is provided, and what users need to do.

hledger-flow's Issues

Change the version subcommand into a --version option

Displaying the version of the command is currently implemented as a subcommand:

$ hledger-makeitso version
hledger-makeitso 0.10.0.0

Change this into a --version flag, with a short option of -V.

The short -V has to be capitalized because a lower-case -v is reserved for a verbose flag.

Read default options from a config file

The config file location should be the basedir, but the location could also be specified via a command-line option.

The config file would support the same options as the command-line options.

In all cases the options specified via the command-line would override any option present in a config file.

Readme doesn't use indentation for Opening/Closing Balance

This example in the readme.org should have indentation.

2018-06-01 Savings Account Closing Balance
assets:Current:MyBank:Savings               $-234.56 = $0.00
equity:Closing Balances:MyBank:Savings

Should be indented like this:

2018-06-01 Savings Account Closing Balance
    assets:Current:MyBank:Savings               $-234.56 = $0.00
    equity:Closing Balances:MyBank:Savings

File-specific rules

On this page, @josephmturner and @adept have proposed to use file-specific rules, i.e. hyperspecific rules that would only apply to one specific imported file, typically because they only target one transaction or transactions from this specific timeframe. The main reason for this is to avoid cluttering the main rules file while still being able to correctly allocate specific transactions, even by overriding general rules in some cases. The second advantage is in terms of performance optimization, as these rules won't be unnecessarily evaluated on every single file.

I think it would make sense for hledger-flow to allow similar file-specific rules and to enforce a way to implement them. In your example directory, I'd see it as:

└── import
    └── gawie
        └── bogart
            └── cheque
                ├── 1-in
                │   └── 2016
                │       ├── 123456789_2016-03-30.csv
                │       └── ...
                ├── 2-preprocessed
                │   └── 2016
                │       ├── 123456789_2016-03-30.csv
                │       └── ...
                ├── 3-journal
                │   └── 2016
                │       ├── 123456789_2016-03-30.journal
                │       └── ...
                ├── bogart-cheque.rules                 # General rules
                ├── ...
                └── rules                               # <-- File-specific rules go here
                    └── 2016
                        └── 123456789_2016-03-30.rules  # File-specific rules, same name as file

Process

Now, as I don't think hledger allows to specify using multiple rules files, we'd have to include the general rules in the specific rules. I'd see the process as following (but maybe there's a better way to do it):

1. On hledger-flow import, check if file-specific rules exist and create them if they don't (don't overwrite) for all detected CSVs. Write the following in each new file:

   include ../../../../../bogart.rules  # if file exists
   include ../../bogart-cheque.rules    # if file exists
   include ../../../../../all.rules     # if file exists (just an idea, that's where I put my generic transactions account-assignment rules, feel free to ignore, as it can also be included manually in any of the above files)
   
   # Write specific transactions rules below this line
   

2. Then, to import, use the file-specific rules only (as they contain all generic rules).

Potential issues

• If there are no rules files whatsoever, there's nothing to include in the specific files, and hledger will complain on an empty rules file. But I suppose that an error message indicating that it cannot import without a rules file would make sense then.
• Maybe it could be difficult for some users to understand where to put their account-identification rules? Or it could incite avoiding generic rules and favoring transaction-specific rules (inefficient)? But I suppose hledger-flow users know better ;-)

Parsing of PDF statements, and multi-currency accounts

Is your feature request related to a problem? Please describe.

My bank only provides CSV/QIF/QFX statements for recent transactions on current accounts, but not savings accounts.
Historic transactions are available as:

• PDFs containing transaction history with running balance
• PDF for individual transaction events between internal/external accounts, called "advice confirms"
• .xls for stock&shares accounts

Luckily the PDFs are very regular, and by using pdfotext and awk I can "infer" a CSV out of it ... most of the time. .xls can be converted to .csv with ssconvert.

I opened this feature request based on https://news.ycombinator.com/item?id=20024633

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.12.2.99 linux x86_64 ghc 8.6

Is this the latest version?

No.

Our Example Statements Repository

FYI, we have a repo with some example transactions which you can use to run hledger-flow on:
https://github.com/apauley/hledger-flow-example

Can you give examples of what you would like by running hledger-flow on these files?
N/A

Describe the solution you'd like

Would be useful to have:

• some way of determining the sign of a transaction based on a field, I currently have to match on DEBIT/CREDIT in my rules file and swap account1 with account2 accordingly. I tried using amount-in and amount-out, but apparently these cannot appear only inside an if statement. Would be nice if there was a conditional in the CSV rules file that could reverse the sign of a transaction.

• support for account3 in csv rules. investment tracking would require 3 accounts, see https://hledger.org/Track-investments.html with just 2 accounts "we have violated the double-entry principle and created money out of nowhere.", however CSV rules only support 2 accounts currently

• some way of recording unrealized gains/loss when handling multiple currencies, see:
  https://github.com/adept/full-fledged-hledger/wiki/Investments-easy-approach#recording-valuations. This would need something that can generate transactions based on prices. See #69 for support of prices

• support fields with default values in CSV rules. See my awk scripts below, often some fields are empty, meaning that they have the same value as the previous non-empty field (this often happens for dates)

• make it easy to record a currency and an exchange rate for multi-currency transactions inside CSV rule conditionals. Currently I have to calculate the final amount in awk, I think I was not able to do this with just CSV rules due to the lack of support for rounding.

• have some built-in support for tracking transactions between 2 of my own accounts, where I have:
  ** an entry in account X saying I did a transaction with reference Z
  ** an entry in account Y saying I did a transaction with reference Z (if an internal account, missing if external)
  ** an entry in my "advice confirms" transaction list saying I transfered something between 2 accounts with reference Z (sometimes ambigous as it doesn't list both account numbers)
  What I currently do here is to have:

current account -> assets:transfers:%ref
assets:transfers:%ref -> assets:transfers:loyisa:%ref
assets:transfers:loyisa:%ref -> assets:isa:loyisa

This is great for debugging why a certain transaction reference is not balanced (e.g. due to a mistake in my import script, or a missing statement), should there be a builtin way of handling these?

I'd welcome any other suggestion you may have on how to simplify the conversion process I've shown below. What I have now works though, took a while to get here though.

Describe alternatives you've considered
Here is a quick and dirty scripts I'm using right now, there's got to be a better way:

$ cat current/preprocess.sh
if [ $(basename -s .pdf "$1") = $(basename "$1") ]; then
    cp "$1" "$2"
else
    pdftotext -nopgbrk -fixed 6 "$1" - | awk -f $(dirname $(realpath "$0"))/preprocess.awk >"$2"
fi

$ cat current/preprocess.awk
BEGIN { OFS=","; FIELDWIDTHS = "9:9 4 1:29 1:15 1:15 1:12" }
/Date.*(Balance|Details)/,/BALANCE CARRIED FORWARD|Summary/ {
    gsub(",","", $0);
    BALANCE = $6;
    if (!(/Balance|Details/)) {
        if (!($1 ~ /^ *$/)) { DATE = $1 };
        if (!($2 ~ /^ *$/)) { TYPE = $2 };
        DESC = DESC " " $3;
        IN = $4;
        OUT = $5;
        if (!(IN ~ /^ *$/ && OUT ~ /^ *$/ && BALANCE ~ /^ *$/)) {
            match(DESC, /ADVICE CONFIRMS +([^ ]*)/, advice)
            ADVICE=advice[1]
            print DATE, TYPE, ADVICE, DESC, IN, OUT, BALANCE
            DESC = ""
        }
    }
}

cat trade/preprocess
#!/bin/sh
set -eu
T=$(mktemp)
ssconvert -S "$1" "${T}.csv"
cat ${T}.csv.* | grep GBP | grep -v Dividend | sed -e 's/GBP/£/' >"$2"

cat cc/preprocess
pdftotext -nopgbrk -fixed 4 "$1" - | awk -f $(dirname $(realpath "$0"))/preprocess.awk >"$2"

TMP=$(mktemp)
pdftotext -layout "$1" - | sed -re 's/ +/ /g' >"$TMP"
(
grep -E "DB Value Date:" "$TMP" | cut -f2 -d:|cut -f2 -d' '
grep -Eo "CR Value Date:.*" "$TMP" | cut -f2 -d:|cut -f2 -d' '
grep -C1 "Account Number" "$TMP" | tail -n1|sed -e 's/^ *//'
cat "$TMP" | grep "your account"|sed -re 's/^.*your account ([^ ]+).*$/\1/'
grep -E "Our Ref:" "$TMP" | cut -f2 -d: |sed -e 's/^ *//'
if grep "ROMANIA" "$TMP" >/dev/null; then
    echo ROMANIA
else
    grep -C1 "Senders Reference:" "$TMP" | tail -n1|sed -e 's/,/ /g' -e 's/^ *//'
fi
grep -C1 "Details of Payment" "$TMP" | tail -n1| sed -e 's/,/ /g'
if grep "ADVICE OF CREDIT" "$TMP" >/dev/null; then
    echo "CREDIT"
    grep -Eo "Amount Credit:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
    grep -Eo "Amount Received:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
elif grep "ADVICE OF DEBIT" "$TMP" >/dev/null; then
    echo "DEBIT"
    grep -Eo "Amount Debited:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
    grep -Eo "Original Amount:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
else
    echo "UNKNOWN TYPE" >&2
    exit 1
fi
grep -Eo "Charges:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
) | sed -e 's/GBP/£/g' -e 's/EUR/€/g'| paste -s -d, >"$2"
ER=$(grep -Eo "Exchange Rate:.*" "$TMP"|cut -f2 -d:|sed -e 's/^ *//')
if grep "ADVICE OF CREDIT" "$TMP" >/dev/null && grep "40XXXXXXXXXXXX" "$TMP" >/dev/null && grep Apr "$TMP" >/dev/null && [ -n "$ER" -a "$ER" != "1.00000000" ]; then
(
   # only have statements for 1 side of the transaction here between 2 accounts
   # have to synthesize the other side based on the exchange rate
    grep -E "DB Value Date:" "$TMP" | cut -f2 -d:|cut -f2 -d' '
    grep -Eo "CR Value Date:.*" "$TMP" | cut -f2 -d:|cut -f2 -d' '
    echo
    echo "40YYYYYYYYYYY DEBIT"
    grep -E "Our Ref:" "$TMP" | cut -f2 -d: |sed -e 's/^ *//'
    echo
    echo autogenerated based on exchange rate
    echo "DEBIT"
    AMOUNT=$(grep -Eo "Amount Credit:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:|sed -e 's/ *GBP *//')
    AMOUNT2=$(zsh -c "zmodload zsh/mathfunc; echo \$(( int($AMOUNT * $ER * 100.0 + 0.5) / 100.0 ))")
    echo "EUR $(printf '%.2f' $AMOUNT2)"
    echo "GBP $AMOUNT"
    grep -Eo "Charges:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
) | sed -e 's/GBP/£/g' -e 's/EUR/€/g' | paste -s -d, >>"$2"
fi
rm "$TMP"

Here is how some the rule file for the above preprocess one looks:

cat xxxx-advice.rules
fields date, _, dest, beneficiary, ref1, ref2, desc, typ, amount1, amount2, charges
date-format %d-%b-%y

amount %amount2

description %ref1 %ref2 %desc

if
CREDIT
    account2 assets:transfers:%ref1

if
40XXXXXXXXXXX.*CREDIT
    account1 assets:current-eur:edwin:xxxx

if
40YYYYYYYYYYYY.*CREDIT
    account1 assets:transfers:current:%ref1

if
40ZZZZZZZZZZZZ.*CREDIT
    account1 assets:transfers:loyisa:%ref1
    account2 assets:transfers:%ref1

if
ROMANIA
    account2 assets:current:edwin:romania

if
DEBIT
    account1 assets:transfers:%ref1

if
40XXXXXXXXXXX.*DEBIT
    account2 assets:current-eur:edwin:xxxx
    amount %amount2 @@ %amount1

Additional context

Generate a recent-years.journal on each level similar

In #5 we created an all-years.journal for each owner, bank and account.

Pointing your LEDGER_FILE to this all-years.journal may become slow for people with many years of input data.

Write a recent-years.journal next to each all-years.journal which contains only the 3 most recent years, where "3" should probably be a command-line option with a sensible default if not specified

Is there a verbose execution mode?

I'd like to see some output when running hledger-makeitso import

Background: hledger-makeitso import repeatedly failed for me when I tried makeitso for the first time today. I eventually figured that I called my preprocess script preprocessor instead of preprocess

Add a README to the release tarball

The release tarballs are generated in the TravisCI builds by running the script bin/release-tarball.

Add a README.org (or README.md) to this release tarball, so that users who download it have some instructions.

The README should briefly mention how to verify the binary (shasum -c sha256-hledger-flow_*) and where to copy it so that it is in your PATH

Handling of historic currency/stock prices

Is your feature request related to a problem? Please describe.

I want to import historic currency exchange rates and stock prices into hledger-flow.
Unfortunately using a construct script doesn't work because hledger-flow filters out and only retains transactions, not prices, see

hledger-flow/src/Hledger/Flow/CSVImport.hs

Line 112 in 14e0150

let args = ["print", "--rules-file", format fp rf, "--file", format fp csvSrc, "--output-file", format fp journalOut]

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.12.2.99 linux x86_64 ghc 8.6

Is this the latest version?

No

Our Example Statements Repository

FYI, we have a repo with some example transactions which you can use to run hledger-flow on:
https://github.com/apauley/hledger-flow-example

Can you give examples of what you would like by running hledger-flow on these files?

No, they do not contain prices.

Describe the solution you'd like
It should be possible to import prices, not just transactions.
First step would be to have an equivalent of the construct script that can output only prices in a format that hledger understands.
A second step would be to have support for csv import rules that can deal with the various date formats and convert to hledger format.

Describe alternatives you've considered

I wrote a script that generates .journal files and puts them into a _manual_ folder.
This is a bit of a hack, because it is not part of the usual hledger-flow import, I have to run it separetely.
What I ended up having is this:

cat import/edwin/_manual_/run.sh
#/bin/sh
set -eu
cd prices;
(for i in $(ls -1d */ | cut -f1 -d/); do
    for j in ${i}/1in/*/*; do
        ${i}/construct "${j}" - X "${i}" X
    done
done) >pricelist
cd ..
for i in 2017 2018 2019; do
    echo 'D £1,000.00' >${i}/pre-import.journal
    grep $i prices/pricelist | sed -e 's/"EURGBP"/€/' >>${i}/pre-import.journal
done

cat import/edwin/_manual_/prices/construct
#!/bin/sh
# this is very slow, should reimplement in python or something else
set -eu
IFS=^
cut -f1-2 --output-delimiter=^ <"$1" | while read D PRICE; do
    D=$(date +%F -d "$D")
    echo P $D \"$4\" $PRICE
done

tail import/edwin/_manual_/prices/EURGBP/1in/2019/EURGBP
Jan 11, 2019 	0.8925 	0.9020 	0.9064 	0.8921 	-1.05%
Jan 10, 2019 	0.9020 	0.9026 	0.9063 	0.9011 	-0.06%
Jan 09, 2019 	0.9025 	0.8997 	0.9044 	0.8973 	0.33%
Jan 08, 2019 	0.8995 	0.8980 	0.9010 	0.8949 	0.18%
Jan 07, 2019 	0.8979 	0.8951 	0.8995 	0.8948 	0.20%
Jan 06, 2019 	0.8961 	0.8952 	0.8967 	0.8949 	0.09%
Jan 04, 2019 	0.8953 	0.9019 	0.9032 	0.8942 	-0.75%
Jan 03, 2019 	0.9021 	0.9000 	0.9133 	0.8986 	0.28%
Jan 02, 2019 	0.8996 	0.8991 	0.9034 	0.8981 	0.07%
Jan 01, 2019 	0.8990 	0.8985 	0.9057 	0.8953 	0.00%

Additional context

Obtaining the prices themselves is another matter, but that is independent of hledger-flow.
What I did was just copy/paste the tables from here, but this could easily be automated by something that fetches the webpage and parses it.

For example investment stock prices can be fetched from (using a random example), you can view 1 year ranges:
https://markets.ft.com/data/funds/tearsheet/historical?s=GB0033053710:GBX

Currencies can be found in a similar place, see:
https://github.com/barrucadu/hledger-scripts/blob/master/market-prices/market-prices.py

Stock prices can be found here (again using a random example)
https://uk.finance.yahoo.com/quote/GOOG/history/?guccounter=1

I haven't done anything interesting with the prices yet, other than being able to see how much I own were I to convert everything into GBP right now, so having daily price granularity is probably an overkill, but it was the easiest way to obtain the data.

`hledger: user error CSVError (ParseErrorBundle...` is rather unclear

(in v0.11.1.2)

I'm experiencing an error importing a CSV, and am at a loss as to whether the problem is in hledger or hledger-flow or my CSV file, or my rules file. The pstateInput = "..." part in the error below contains my whole CSV input file, which I cannot quote here because it is huge and contains PII, so i just put ...

2019-05-08 15:10:48.668158024 EEST	hledger-flow End:   importing '2-preprocessed/2018/2018.csv' using rules file 'bank-account.rules' ExitFailure 1 (0.081442287s)
hledger: user error (CSVError (ParseErrorBundle {bundleErrors = TrivialError 14 (Just (Tokens (59 :| []))) (fromList [Tokens (44 :| []),Label ('e' :| "nd of line"),EndOfInput]) :| [], bundlePosState = PosState {pstateInput = "...", pstateOffset = 0, pstateSourcePos = SourcePos {sourceName = "import/personal/bank/account/2-preprocessed/2018/2018.csv", sourceLine = Pos 1, sourceColumn = Pos 1}, pstateTabWidth = Pos 8, pstateLinePrefix = ""}}))

hledger-flow: an external process exited with exit code 1.

The CSV file in question loads without error in Libreoffice

In order to rule out whether it's a quoting issue, or a separating character issue, I'd need to know what quoting character and / or separating character was in use. The error and log statement do not say.

The (probably incomplete) rules file:

skip 7

fields Buchungstag,Wertstellung,Buchungstext,Auftraggeber_Beguenstigter,Verwendungszweck,Kontonummer,BLZ,Betrag_EUR,Glaeubiger_ID,Mandatsreferenz,Kundenreferenz

account1 Personal:Assets:DE:Bank:Account

amount EUR %Betrag_EUR

date %Wertstellung
date-format %d.%m.%Y

currency EUR

There are also non ascii characters, but that should be no problem in a UTF8 file

<LATIN SMALL LETTER A WITH DIAERESIS> -> ä
<LATIN SMALL LETTER O WITH DIAERESIS> -> ö
<LATIN SMALL LETTER U WITH DIAERESIS> -> ü

hledger-flow does not 'see' _manual_ year subdirectory if there is no corresponding 1-in subdirectory

Describe the bug

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ $HLEDGER_FLOW_EXECUTABLE --version
hledger-flow 0.15.0 linux x86_64 ghc 9.0 2b025fee477538a493cb7b5eaf163fd959aaec42

Is this the latest version?

Yes

Also include the runtime options of the command you ran, e.g:

/home/lestephane/.local/bin/hledger-flow-0.15.0 +RTS -N10 -RTS --show-options --batch-size 200 import
RuntimeOptions {baseDir = "/home/lestephane/Vault/Finance/", importRunDir = "import/personal/SCRUBBED/fondspolice/", importStartYear = Nothing, onlyNewFiles = False, hfVersion = "hledger-flow 0.15.0 linux x86_64 ghc 9.0 2b025fee477538a493cb7b5eaf163fd959aaec42", hledgerInfo = HledgerInfo {hlPath = "/home/lestephane/Vault/Finance/hledger", hlVersion = "hledger 5895a46, linux-x86_64"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [9,0], versionTags = []}, cores = 12, availableCores = 10}, verbose = False, showOptions = True, sequential = False, batchSize = 200}
Collecting input files from /home/lestephane/Vault/Finance/import/personal/SCRUBBED/fondspolice/
Found 1 input files in 0.000833711s. Proceeding with import...
Wrote include files for 1 journals in 0.002329759s
Imported 1/1 journals in 0.039248629s

To Reproduce

• Import a directory for which
  • There is a 1-in journal for year YYYY
  • There is no 1-in journal for year YYYY+1 << -- this part I'm not sure about
  • There is a _manual_/YYYY+2/post-import.journal

Expected behavior

• There should be a YYYY+2-include.journal
• The YYYY+2-include.journal should be included in all-years.journal

Desktop (please complete the following information):
N/A

Additional context
N/A

hledger-flow refuses to process statements in the year 2011

I've issued many hledger-flow import commands yesterday, and it could well be that one of them created include files where it shouldn't have. Now i'm unable to import.

~/Finance/import/personal/somebank$ hledger-flow import --sequential
Collecting input files...
Found 12 input files in 0.054998203s. Proceeding with import...
I couldn't find the right number of directories between "import" and the input file:
.../Finance/import/personal/somebank/depot/2011-include.journal

hledger-flow expects to find input files in this structure:
import/owner/bank/account/filestate/year/trxfile

Have a look at the documentation for a detailed explanation:
https://github.com/apauley/hledger-flow#input-files

I don't think it's worth it to figure out (just yet) how I got there.

But my question is: how do I safely remove all the include files generated by previous hledger-flow import runs, so that I may start fresh, from a known good state? Is the command below a good first approximation?

find . -type f | grep -E "[0-9]{4}-include.journal|all-years.journal" | xargs rm

Move the verbose flag from the subcommands to the main list of options

Related to #14

The --verbose flag is currently enabled on each subcommand, e.g.:

$ hledger-flow import --verbose
2019-04-27 11:03:05.808182368 SAST	hledger-flow Starting import

or

$ hledger-flow report --verbose
2019-04-27 11:03:33.90158193 SAST	hledger-flow Something will be here Real Soon Now (tm)

Change this so that the verbose flag is part of the main options, similar to --help and --version.

However, unlike --help and --version, the verbose flag should not just print some output and exit.
Instead, it should continue to expect one of the subcommands to be supplied, and then run the relevant subcommand with verbose output enabled.

If I delete a file in a `1-in` directory, re-running `hledger-flow import` does not remove the corresponding files in the `2-preprocessed` and `3-journal` directories

First off, thanks for a wonderful tool!

Describe the bug
If I delete a file in a 1-in directory, re-running hledger-flow import does not remove the corresponding files in the 2-preprocessed and 3-journal directories.

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.14.4 linux x86_64 ghc 9.0 UNKNOWN

I know this is not the latest version of hledger-flow, sorry. I looked at the diff between 0.14.4 and 0.15, and I don't think anything related to this has been changed. Let me know if it's really important to test with the latest version of hledger-flow, and I'll figure out how to update.

Also include the runtime options of the command you ran, e.g:

$ hledger-flow --show-options import

To Reproduce

I can reproduce this with a fresh clone of https://github.com/apauley/hledger-flow-example:

$ rm import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv

$ hledger-flow import . --show-options
RuntimeOptions {baseDir = "/home/jeremy/src/github.com/apauley/hledger-flow-example/", importRunDir = "./", importStartYear = Nothing, onlyNewFiles = False, hfVersion = "hledger-flow 0.14.4 linux x86_64 ghc 9.0 UNKNOWN", hledgerInfo = HledgerInfo {hlPath = "/nix/store/lxv7a7ps6k7girbwm50kr3w42ih90v8w-hledger-1.25/bin/hledger", hlVersion = "hledger 1.25, linux-x86_64"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [9,0], versionTags = []}, cores = 4, availableCores = 4}, verbose = False, showOptions = True, sequential = False, batchSize = 200}
Collecting input files from /home/jeremy/src/github.com/apauley/hledger-flow-example/import/
Found 2 input files in 0.001297681s. Proceeding with import...
Wrote include files for 3 journals in 0.002288837s
Imported 2/2 journals in 0.061123857s

$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	deleted:    import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv

no changes added to commit (use "git add" and/or "git commit -a")

Expected behavior

In the example above, I'd expect import/gawie/bogart/cheque/2-preprocessed/2016/123456789_2016-03-30.csv and import/gawie/bogart/cheque/3-journal/2016/123456789_2016-03-30.journal to get deleted as well.

For now, I am working around this by first nuking my 2-preprocessed and 3-journal directories before doing an import:

$ rm -r import/*/*/*/2-preprocessed import/*/*/*/3-journal
$ hledger-flow import .

Desktop (please complete the following information):
I'm running Arch Linux. I installed hledger and hledger-flow with the Nix package manager.

The -closing include should be part of all-years.journal, not yyyy-include.journal

Description

In the situation where one has yyyy-opening.journal and yyyy-closing.journal for a specific year,
running one of the hledger built-in financial reports on one specific year, the user will be faced with empty reports

$ hledger bal 2017-include.journal 
--------------------
                   0

because hledger-flow adds an include to yyyy-closing.journal in the yyyy-include.journal, which effectively brings all the counters back to zero.

Expectation

I would expect hledger <any report> -f yyyy-include.journal to just work

Solution

The yyyy-opening.journal include stays in yyyy-include.journal, but yyyy-closing.journal include needs to move to the all-years.journal file, as evidenced by the all.journal in the full-fledged-hledger project. This ensures that reports work on a per-year yyyy-include.journal basis, and that reports also work on an aggregated all-years.journal basis, with the important caveat that the very last yyyy-closing.journal must not be part of the all-years.journal (even if one such file is present on the file system), otherwise we encounter the same problem of empty reports on an aggegated all-years.journal basis.

I hope that makes sense, it's a bit tricky to explain.

Version and Runtime Information

This include behaviour has been present in hledger-flow as far back as I can remember using it.

Additional context

I only found this bug because I'm trying to use hledger-flow import for my imports, while using full-fledged-hledger's export.hs for my reports, and I got confused that I was getting empty reports.

Proposal: persisting the autopostings to journal files would allow for review of changes made to rewriting rules

Is your feature request related to a problem? Please describe.

I use autoposting rules to keep track of some out-of-pocket expenses that my company owes me.
If I make a change to an autoposting rule, I would like to review the ripple effect of the change, and of course version control it (not just the rule, all affected postings). If a change has many ripple effects, I'm much more likely to spot a problem at the gif diff stage. Currently I only see a problem when running hledger-ui after painfully looking for the transaction. If there is a problem with hledger itself (as when the synthetic transaction contains amounts that are formatted in an ambiguous way), I can't do anything, as the transaction does not 'exist' on the filesystem. This is how for example, a sum of 2.500 EUR from an autoposting gets interpreted as 2.5 EUR and I can't do anything about it, I can only specify a multiplier (*1, *-1, etc). If I could at least have these autopostings generated and visible in a file somewhere, i could apply an additional regex based search and replace in my build pipeline.

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.12.3.0 linux x86_64 ghc 8.6

Is this the latest version? YES

If your request includes commands you ran and the output, please also include
the runtime options with --show-options e.g:

Describe the solution you'd like

• one new file that is looked for by hledger-flow (besides construct and preprocess) -> rewrite-rules (name inspired from https://hledger.org/manual.html#re-write-rules-in-a-file's rewrite-rules.journal file parameter example)
• hledger-flow includes import works in all respect as before, except...
• ... if rewrite-rules is found, the journal generation command for 3-journal includes
  • the --auto command-line option
  • an additional -f command-line option pointing to the rewrite-rules
• rewrite-rules is not included in YYYY-include.journal, since the 3-journal files then already contain the side-effects of applying the rewriting rules.

(Note: I don't know how the 3-journal files are generated today, and I can't see any 3-journal related output in the verbose logs, so I indicated how I would do it using command-line options to the hledger print command)

Describe alternatives you've considered

The way I would do this today is by invoking

$ hledger-flow import
$ hledger print --auto > before.txt
(make a change to autoposting rules)
$ hledger print --auto > after.txt
diff before.txt after.txt

If the aggregated journal is rather large, it will become impractical to run a diff on it.

Potential problems
If rewrite rules are put in some other shared include file (à la #50) and not rewrite-rules files, hledger-flow might not detect that --auto needs to be added when generating the journals.

The behaviour is undefined in that case. The user is expected to put autopostings in a file by that name, which should standardize things. Since hledger-flow is opinionated anyway, that should be fine.

Smart date should include "last WEEKDAY" and "X days ago"

I would use those a lot...

Edit:
I am a new user and enter all data with hledger add (on the terminal).
At times I don't ask for a receipt because I can just memorize the price or it's just one item that I buy often.
~4 days later I remember, hey didn't I buy X last week? I think it was Thursday...

What I currently do:
hledger add
open external calendar see which date "last Thursday" is and copy the date into the hledger add prompt.

I'd like to do:
hledger add
Date [2019/04/23]: last thursday

I just realized that tab fills in the current date, so being able to type in 4 days ago isn't that urgent, but since yesterday works I expect other phrases to work and with tab completion as for yesterday l[ast] tab th[ursday] tab and 4 d[ays] tab a[go] tab would feel more convenient to me.

QUESTION: tags, reports, multiple contributors, virtual accounts, how to do it simply?

Is your feature request related to a problem? Please describe.

Hi, this is a great project! This has helped me make full-fledged-hledger (and hledger!) significantly more accessible, thank you for that.

Being new to hledger/PTA and hledger-flow, I may just be overlooking how to do what I want to do with existing features. If that is the case, I apologize and would appreciate guidance on what to do.

There are a couple things I would like to do which may or may not be related, so I'll throw them out there together:

• hledger seems to support tags, and they seem useful (maybe not?). Can I use them with hledger-flow and csv imports? I tried defining the field in CSV import rules, but it did not seem to work out ok.
• I have a bunch of "projects" each with their own (virtual) accounts and expenses, liabilities, etc, should I keep them in my primary hledger-flow import or should I separate the projects in some fashion? Maybe you have other recommendations for these types of "virtual" accounts?
• I have some use-cases where an "invoice" or "bill" comes in, and then "multiple parties" contribute to those liabilities, though the actual payment of the invoice/bill is separate. I am having a difficult time understanding the clear/obvious way to track this in hledger-flow. For example, when the invoice is received, or a contribution is made, or the bill is actually paid, I am not sure what to put in my CSV for the accounts.
• If I have different expenses go to expenses:payables:Acme Corp and expenses:payables:Pizza Shop and such, but would also like to say those are associated with a particular project or tag in reports, should I be using tags or some other tricks with accounts?

Version and Runtime Information

hledger-flow 0.13.2.0

hledger-flow fails path tests on windows systems

Describe the bug
When building hledger-flow in a Windows (Windows10) system, the executable is built correctly, but the tests fail, because of the path-separator. The included tests expect a unix path separtor / while the system returns the Windows path separator \.
This is only a thing in the tests; the code itself checks for the path-separator.

Installing and running the hledger-app and importing data works as expected

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.13.2.0 mingw32 x86_64 ghc 8.8

This is the last git version I'm running (just updated)

$ hledger-flow --show-options import
RuntimeOptions {baseDir = FilePath "C:\\Users\\max\\source\\hledger-flow-example\\", importRunDir = FilePath ".\\", useRunDir = False, hfVersion = "hledger-flow 0.13.2.0 mingw32 x86_64 ghc 8.8", hledgerInfo = HledgerInfo {hlPath = FilePath "C:\\Users\\max\\AppData\\Roaming\\local\\bin\\hledger.exe", hlVersion = "hledger 1.17.1.1"}, sysInfo = SystemInfo {os = "mingw32", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [8,8], versionTags = []}}, verbose = False, showOptions = True, sequential = False}
Collecting input files from C:\Users\max\source\hledger-flow-example\import\
Found 3 input files in 0.0079695s. Proceeding with import...
hledger-flow: C:\Users\max\source\hledger-flow-example\import\gawie\bogart\cheque\preprocess: createProcess: invalid argument (Exec format error)

To Reproduce
Can you reproduce your issue on these example files?
I was using the example files when running the --show-options import command above.

Steps to reproduce the behavior:

1. Download the code per instructions on the contributing guidelines
2. Then ran the command stack test --interleaved-output --pedantic
3. And got the following test error:

Registering library for hledger-flow-0.13.2.0..
hledger-flow> test (suite: hledger-flow-test)

### Failure in: 1:0
test\Common\Integration.hs:94
determineBaseDir produces an error message when given a non-existant dir
expected: Left "The provided directory does not exist: /path/to/dir"
 but got: Left "The provided directory does not exist: \\path\\to\\dir"
### Failure in: 2:7
test\CSVImport\Unit.hs:424
Convert a grouped map of paths, to a map with text contents for each file
expected: fromList [(FilePath ".\\import\\john\\bogartbank\\checking\\2018-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import/john/bogartbank/checking/3-journal/2018/2018-10-30.journal\n!include import/john/bogartbank/checking/3-journal/2018/2018-11-30.journal\n!include import/john/bogartbank/checking/3-journal/2018/2018-12-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\checking\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import/john/bogartbank/checking/3-journal/2019/2019-01-30.journal\n!include import/john/bogartbank/checking/3-journal/2019/2019-02-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\savings\\2017-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import/john/bogartbank/savings/3-journal/2017/2017-11-30.journal\n!include import/john/bogartbank/savings/3-journal/2017/2017-12-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\savings\\2018-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import/john/bogartbank/savings/3-journal/2018/2018-01-30.journal\n!include import/john/bogartbank/savings/3-journal/2018/2018-02-30.journal\n")]
 but got: fromList [(FilePath ".\\import\\john\\bogartbank\\checking\\2018-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import\\john\\bogartbank\\checking\\3-journal\\2018\\2018-10-30.journal\n!include import\\john\\bogartbank\\checking\\3-journal\\2018\\2018-11-30.journal\n!include import\\john\\bogartbank\\checking\\3-journal\\2018\\2018-12-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\checking\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import\\john\\bogartbank\\checking\\3-journal\\2019\\2019-01-30.journal\n!include import\\john\\bogartbank\\checking\\3-journal\\2019\\2019-02-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\savings\\2017-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import\\john\\bogartbank\\savings\\3-journal\\2017\\2017-11-30.journal\n!include import\\john\\bogartbank\\savings\\3-journal\\2017\\2017-12-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\savings\\2018-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import\\john\\bogartbank\\savings\\3-journal\\2018\\2018-01-30.journal\n!include import\\john\\bogartbank\\savings\\3-journal\\2018\\2018-02-30.journal\n")]
### Failure in: 3:2
test\CSVImport\Integration.hs:95
All pre/post files on disk should be included
expected: fromList [(FilePath ".\\hlflow-1e1497e36b7682f9\\import\\john\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include _manual_/2019/pre-import.journal\n!include bank1/2019-include.journal\n!include bank2/2019-include.journal\n!include _manual_/2019/post-import.journal\n")]
 but got: fromList [(FilePath ".\\hlflow-1e1497e36b7682f9\\import\\john\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include _manual_\\2019\\pre-import.journal\n!include bank1\\2019-include.journal\n!include bank2\\2019-include.journal\n!include _manual_\\2019\\post-import.journal\n")]
### Failure in: 3:3
test\CSVImport\Integration.hs:119
All opening/closing files on disk should be included
expected: fromList [(FilePath ".\\hlflow-c14016fa77f9a37e\\import\\john\\bank1\\savings\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include 2019-opening.journal\n!include 3-journal/2019/2019-01-30.journal\n!include 2019-closing.journal\n")]
 but got: fromList [(FilePath ".\\hlflow-c14016fa77f9a37e\\import\\john\\bank1\\savings\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include 2019-opening.journal\n!include 3-journal\\2019\\2019-01-30.journal\n!include 2019-closing.journal\n")]
### Failure in: 3:4
test\CSVImport\Integration.hs:144
The price file should be included together with any pre/post files
expected: fromList [(FilePath ".\\hlflow-72de84936923ac78\\import\\2020-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include _manual_/2020/pre-import.journal\n!include john/2020-include.journal\n!include ../prices/2020/prices.journal\n!include _manual_/2020/post-import.journal\n")]
 but got: fromList [(FilePath ".\\hlflow-72de84936923ac78\\import\\2020-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include _manual_\\2020\\pre-import.journal\n!include john\\2020-include.journal\n!include ..\\prices\\2020\\prices.journal\n!include _manual_\\2020\\post-import.journal\n")]
### Failure in: 3:5
test\CSVImport\Integration.hs:198
John1: The include file contents should be the journal files
expected: "### Generated by hledger-flow - DO NOT EDIT ###\n\n!include 3-journal/2018/2018-10-30.journal\n!include 3-journal/2018/2018-11-30.journal\n!include 3-journal/2018/2018-12-30.journal\n"
 but got: "### Generated by hledger-flow - DO NOT EDIT ###\n\n!include 3-journal\\2018\\2018-10-30.journal\n!include 3-journal\\2018\\2018-11-30.journal\n!include 3-journal\\2018\\2018-12-30.journal\n"
Cases: 20  Tried: 20  Errors: 0  Failures: 6

hledger-flow> Test suite hledger-flow-test failed
Completed 2 action(s).
Test suite failure for package hledger-flow-0.13.2.0
    hledger-flow-test:  exited with: ExitFailure 1
Logs printed to console

Expected behavior
Pass the test

Desktop (please complete the following information):

• OS: Windows10 Pro
• Version 1909

Additional context
Add any other context about the problem here.

Question: where to include "meta" statements (`account...`, `commodity format` & `alias`) & prices?

First, thank you for hledger-flow! I really appreciate the opinionated guidance on structure and the help on the non-destructive workflow.

I have a really basic question, but which I really couldn't find any answer about, nor here, nor on hledger-flow-example, nor on full-fledged-hledger:

Where do you put "meta"/generic statements that apply to all your hledgering, such as:

account assets:bank:savings
account liabilities:bank:creditcard

alias USD = $
commodity 
  format  1,000.00 $
D 1,000.00 $

include import/prices/YEAR.prices

In my initial, single-file journal workflow, I had a meta.journal which included all this and was included from the main journal, but here, I'm struggling to find where it would be appropriate to include it.

Do you use a main journal file besides those created by hledger-flow?
Where does LEDGER_FILE env constant points to? (= where does hledger defaults to without -f?)
That seems important to be able to use hledger without -f or outside its main directory, and so that you can use hledger add to add manual transactions directly to the right place (in owner/_manual_/2021/post-import.journal, I guess?)

---

Note: I think this basic question (and other similar ones) could (have) be(en) solved by adding more files to hledger-flow-example: a _manual_ directory with some basic examples in pre-import.journal and post-import.journal (I'm not sure what goes in each), an import/prices/2016.prices example and where it is included, some file with the meta statements I'm mentioning, etc. (If it can help you, I can do a PR with these files as soon as I understand the recommended practice on the matter.)

I'd ideally also include on hledger-flow-example a .gitignore file: I'm guessing it is not really necessary to version control files that are entirely generated and transitional (e.g., 2-preprocessed/...)? Do you actually version control both source (1-in/*.csv) and generated (3-journal/*.journal) files?

And a README listing some typical commands you would run within this workflow could be useful as well :-)

Allow multiple levels of verbosity

Related to #14, #39 and #1

Allow multiple occurrences of the -v flag to indicate increased levels of verbosity.

Example:

$  hledger-flow report -vvv

How to deduplicate 2 transactions that are really one transfer from one bank account to another

in bank1.journal, imported from bank1.csv

2018/11/09 (6138507615589995706)
    Personal:Assets:BANK1:Checking:EUR      EUR -11,22 = EUR 33,44
    Personal:Assets:BANK2:Girokonto:EUR

in bank2.journal, imported from bank2.csv

2018/11/09 (6138507615589995706)
    Personal:Assets:BANK2:Girokonto:EUR        EUR11,22
    Personal:Assets:BANK1:Checking:EUR       EUR-11,22

When I then run hledger-ui, I get an balance assertion error, because it sees two debits of EUR-11,22, not a single one, within the same day. I added the transaction code to try to make hledger-ui see one unique transaction, to no avail. It seems to work that way by design, which makes it difficult to handle transfers between my accounts in the hledger-flow world.

I tried to use virtual postings for bank2, to no avail.

What is the correct way to do this in the hledger world, and most importantly in the hledger-flow world?

Approach 1: Remove problematic line in the bank2 preprocess script

Probably the simplest approach, the bank2 preprocess script could have a grep -v <txid we want to ignore>, so that the transaction ends up in only one journal.

Problem: In the bank subdirectory, there is a CSV line for the transaction in question in 1-in, but no corresponding transaction in 3-journal. It feels like we're rewriting history.

Additionally, we now lose some information, such as when the day on which the money leaves bank1 is different from the day on which the money arrives in bank2.

Approach 2: Comment out the problematic journal entry using a bank2 construct script

Problem: You need to switch to using a construct file, you lose the ability to use a .rules file.

---

My current approach is Approach 1, since in my case bank2 uses a .rules file.

Have you encountered this situation, and how did you resolve it?

hledger-flow does not 'see' hledger despite it being present in the PATH as a symlinked executable

To Reproduce

$ cd import
$ export PATH=".:$PATH"
$ ln -s ~/.local/bin/hledger-flow-async-batches-793f882bb22ac7b89a98077ee95b3464bbc5c0e0 hledger-flow
$ ln -s ~/.local/bin/hledger-32b42a6c8082e2567a73365d48696354f895c2f9 hledger
$ hledger-flow import
Unable to find hledger in your path. # <<- no time to investigate that one, it should have worked
You need to either install hledger, or add it to your PATH, or provide the path to an hledger executable.
$ which hledger
./hledger
$ hledger --version
hledger 1.21.99

hledger-flow should exit with a non-zero error code if a journal construct invocation results in ExitFailure

Background
I have PDF statements from my bank which i convert using a construct script (combination of pdftotext and awk script).

During the hledger-flow import invocation, I see some error, and indeed the corresponding journal is never created.

Collecting input files...
Found 23 input files in 0.134708928s. Proceeding with import...
...
hledger: -:161:1:
    |
161 | 2018-02-30 "Abschluss lt. Anlage 1 PN:905"
    | ^^^^^^^^^^
well-formed but invalid date: 2018-2-30

2019-04-15 17:04:58.505274683 EEST	hledger-flow End:   executing 'girokonto/construct' on 'girokonto/1-in/2018/Kontoauszug_vom_28.02.2018_05.03.2018.pdf' ExitFailure 1 (0.082273261s)
...
Imported 23 journals in 2.203744436s

the problem is twofold:

• there is now a YYYY-include.journal file that contains the !include of a file that does not exist on the filesystem in 3-journal
• the importcommand did not return a non-zero exit code (despite the construct ExitFailure 1). So, one will only see a problem later (potentially days later), when loading the files with hledger. By then the user won't know what made his file disappear.

$ hledger-ui -f all-years.journal 
...
in file included from /Finance/import/personal/bbbank/2018-include.journal,
/Finance/import/personal/bbbank/girokonto/2018-include.journal:4:97:
  |
4 | !include 3-journal/2018/Kontoauszug_vom_28.02.2018_05.03.2018.journal
  |                                                                                                 ^
No existing files match pattern: 3-journal/2018/Kontoauszug_vom_28.02.2018_05.03.2018.journal

Move the Feature Reference to Read The Docs

Move the Feature Reference out of our README, and into
https://hledger-flow.readthedocs.io/

Also update the docURL function to point to the new URL, and ensure that existing usages will generate valid links.

hledger-flow/src/Hledger/Flow/Common.hs

Line 234 in 1729200

docURL = format ("https://github.com/apauley/hledger-flow#"%l)

hledger executable not found on windows

Describe the bug
The hledger executable is not found by hledger-flow on windows. It tries to search for the executable file hledger in the path, but on windows it should look for hledger.exe.

Changing the file it searches for in src\Common.hs on line 59 from maybeH <- which "hledger" to maybeH <- which "hledger.exe" fixes the problem on windows, so a platform check and saerch accordingly would be solution here.

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.13.2.0 mingw32 x86_64 ghc 8.8

This is the last git version I'm running (just updated)

Desktop (please complete the following information):

• OS: Windows10 Pro
• Version 1909

Add a command-line option to suppress all output (--quiet)

Add a --quiet switch which would suppress all output, except for failures causing the program to exit.

This is incompatible with --verbose, if both are specified we should exit with an error message.

verbose execution output looks mangled

I've tried the verbose execution mode flag (-v) and the output looks really weird:

$ hledger-flow import -v
2019-04-13 08:57:52.983225837 EEST	hledger-flow Starting import
Collecting input files...
2019-04-13 08:57:53.091685212 EEST	hledger-flow Found 2 input files in 0.10839391s. Proceeding with import...
2019-04-13 08I: 5c7o:u5l3d.n0'9t1 7f5i3n7d7 4a nE EhSlTe	
... more garbage)

I cannot add the full output as, if later demangled, if could contain personal information.
If you need the full output, i can provide it to the author using send.firefox.com

Some notes about my setup:

• Ubuntu 16.04
• hledger-flow 0.11.0.0
• home directory is ecrypfs encrypted
• additionally, the directory i run the command in is encfs encrypted
• in all other respects, the command runs fine (with or without -v), just the output is screwed
• ghc 7.10.3
• stack version 1.9.3

when preprocess is called with a $1 that has a .timeclock extension, $2 has a .csv extension

Describe the bug
I use .timeclock files in one of my time 'account' directories where there is a preprocess and a construct script.
The preprocess script is called with $1 = $filename.timeclock and $2 = $filename.csv. The problem is that my
preprocess script does not alter the format of the file, it just cleans it. I end up with a csv file that is still a timeclock
file, but has a .csv extension.

This triggers an error when in the construct script, i then make use of that file in an !include

$ cat construct 
#!/usr/bin/env bash

source common.sh

hledger print -f - <<EOF
$(<"${my_dir}/aliases.journal")
!include ${1}
EOF

when the construct script is run with $1 = $filename.csv

hledger-1.20.2: sorry, CSV files can't be included yet
caught ERR (exit code : 1)

This behaviour did not cause any error in the past, hledger would just figure out from the content file what format it's in.
Now the behaviour has changed, and hledger will complain.

I guess this falls under the discussion of should preprocess always result in a CSV file. Preprocess could well be
a cleanup step that does not change the extension of the file, but then how to signal this?

The workaround is for me to pack all the steps in a single construct file.

Reproduction

• Create a .timeclock file in 1-in
• Create a process script that only cats $1 to $2
• Create a construct script that feeds !include $1 to hledger print -f -
• Error: CSV files can't be included yet

Version and Runtime Information

RuntimeOptions {baseDir = "/home/lestephane/Vault/Finance/", importRunDir = "import/special/timespent/hours-in-project/", importStartYear = Nothing, onlyNewFiles = False, hfVersion = "hledger-flow 0.14.2.0 linux x86_64 ghc 8.10", hledgerInfo = HledgerInfo {hlPath = FilePath "/home/lestephane/Finance/hledger", hlVersion = "hledger 1.20.2"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [8,10], versionTags = []}}, verbose = True, showOptions = True, sequential = True}

Need a way to import only the current year

Is your feature request related to a problem? Please describe.

As my full import takes longer and longer, it starts making sense to restrict the years to import. Currently it imports as far back as it finds years. My top 5 time consumers during full import are from past year.

I would need to import only the current year, and I could revert to full import when I make
a change that influences past journal generation, but only pay that extra time when I do the full import.

I'm reluctant to put actual stats here, because my files contain personal info, but If you need it I can give you an idea of the top 5, which each take a couple minutes, and are all before 2020.

Document a GitHub Actions config to validate hledger-flow

Not really a feature request, but not sure where to document this.

Below is a setup for GitHub Actions, to validate a repository with hledger-flow

# .github/workflows/hledger-flow.yml

name: Validate hledger-flow

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1

    - name: Install hledger
      run: docker pull dastapov/hledger

    - name: Install hledger-flow
      run: curl -L https://github.com/apauley/hledger-flow/releases/download/v0.12.4.0/hledger-flow_Linux_x86_64_v0.12.4.0_4b9b027.tar.gz | tar xvz && mv hledger-flow_Linux_x86_64_v0.12.4.0_4b9b027/hledger-flow .

    - name: Grant permissions to create files
      run: chmod 777 -R ./my-finances

    - name: Test hledger file
      run: docker run --name="ledger" -v $(pwd):/data dastapov/hledger ./hledger-flow import ./my-finances

where to put account aliases and type declarations (aka shared journal fragments needed by all journals)?

(in v0.11.1.2)

I need to define two account aliases for accounts that I now use in multiple preprocess and construct files, to shield myself from having to modify multiple locations if I ever need to rename those two accounts (a distinct possibility)

alias aws:personal = Personal:Expenses:Admin:Hosting:AWS
alias aws:businessone = BusinessOne:Expenses:Admin:Hosting:AWS

Problem: I don't see the option to create a journal file that is included by all journal files.

There is a all-years.journal at the top-level, but that file is generated.

Without support from hledger-flow for adding a global journal, I don't see how to achieve this cleanly.

I'm going to try to use a yearly pre-import that is shared in multiple year subdirectories using symlinks, and will report whether that is an acceptable workaround.

btw, I tried running in --verbose mode, but the log output does not report any attempt to look for manual pre / post import journal. Anyway, I will try, and will report.

$ hledger-flow import --verbose 2>&1 | grep -E "_manual_|pre-import|post-import"
(no output)

hledger-flow can only be run in same folder as import folder

Describe the bug
A clear and concise description of what the bug is.

Lowest level auto generated files use relative import starting from the top import path. All other autogenerated journal files use the correct relative path. This does not happen is hledger-flow is started in the same folder that import is defined.

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.12.3.0 linux x86_64 ghc 8.6

Also include the runtime options of the command you ran, e.g:

[max@thinki flow]$ ./hledger-flow --show-options import ledgers/
RuntimeOptions {baseDir = FilePath "ledgers/", hfVersion = "hledger-flow 0.12.3.0 linux x86_64 ghc 8.6", hledgerInfo = HledgerInfo {hlPath = FilePath "/usr/bin/hledger", hlVersion = "hledger 1.14.2"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [8,6], versionTags = []}}, verbose = False, showOptions = True, sequential = False}
Collecting input files...
Found 2 input files in 0.004298132s. Proceeding with import...
Imported 2 journals in 0.270081829s

To Reproduce

1. Download the example repository
2. ./hledger-flow import hledger-flow-example
3. ./hledger-flow report hledger-flow-example

=== Begin Error: hledger --file all-years.journal --period 2016 balance --pretty-tables --quarterly --flat --no-total transfer ===
External command:
/usr/bin/hledger --file hledger-flow-example/all-years.journal --period 2016 balance --pretty-tables --quarterly --flat --no-total transfer
Exit code 1
Error output:
hledger: in file included from /home/max/test/hledger-flow-example/all-years.journal,
in file included from /home/max/test/hledger-flow-example/import/2016-include.journal,
in file included from /home/max/test/hledger-flow-example/import/gawie/2016-include.journal,
in file included from /home/max/test/hledger-flow-example/import/gawie/bogart/2016-include.journal,
/home/max/test/hledger-flow-example/import/gawie/bogart/cheque/2016-include.journal:4:101:
  |
4 | !include hledger-flow-example/import/gawie/bogart/cheque/3-journal/2016/123456789_2016-03-30.journal
  |                                                                                                     ^
No existing files match pattern: hledger-flow-example/import/gawie/bogart/cheque/3-journal/2016/123456789_2016-03-30.journal


=== End Error: hledger --file all-years.journal --period 2016 balance --pretty-tables --quarterly --flat --no-total transfer ===

Expected behavior

I can run hledger-flow in a different folder as specified in the documentation.

Desktop (please complete the following information):

$ uname -a
Linux thinki 5.1.4-arch1-1-ARCH #1 SMP PREEMPT Wed May 22 08:06:56 UTC 2019 x86_64 GNU/Linux

Log the exit status of each shell command

Verbose logs for the external script hooks and other shell commands currently do not log the exit status of the command:

2019-04-05 23:27:36.499626412 SAST	hledger-makeitso Begin: executing 'import/gawie/bogart/cheque/preprocess' on 'import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv'
2019-04-05 23:27:36.531189098 SAST	hledger-makeitso End:   executing 'import/gawie/bogart/cheque/preprocess' on 'import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv' (0.031460856s)

Log the exit code as well, so that the logs look something like:

2019-04-05 23:27:36.499626412 SAST	hledger-makeitso Begin: executing 'import/gawie/bogart/cheque/preprocess' on 'import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv'
2019-04-05 23:27:36.531189098 SAST	hledger-makeitso End:   executing 'import/gawie/bogart/cheque/preprocess' on 'import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv' ExitSuccess (0.031460856s)

This will probably require us to change usages of procs into proc.

Documentation that may prove useful:
https://hackage.haskell.org/package/turtle-1.5.13/docs/Turtle-Prelude.html#v:proc

Make it possible to configure the number of cores being used (the default is to use all cores, which slows down the machine)

I have 8 cores (4 physical + hyperthreading), and this is what it looks like when I run hledger-flow --import

Can we have a way to change the default? I may want to run only with 2 cores if there are a lot of open applications...

Add a command-line option to disable parallel processing

If the option is specified the program will import statements sequentially. The same for report generation.

Parallel processing should still be the default behaviour.

`hledger-flow report` fails with latest hledger release (1.15.2) because of `Flag requires argument: --value`

$ hledger-flow-v0.12.3.1 --version
hledger-flow 0.12.3.1 linux x86_64 ghc 8.6
$ hledger --version
hledger 1.15.2
$ hledger-flow-v0.12.3.1 report --sequential --verbose
Report generation is still a work-in-progress - please let me know how this can be more useful.

Keep an eye out for report-related pull requests and issues, and feel free to submit some of your own:
https://github.com/apauley/hledger-flow/pulls
https://github.com/apauley/hledger-flow/issues

2019-10-13 12:39:24.644119269 EEST	hledger-flow Begin: hledger --file all-years.journal --period 2017 balance --pretty-tables --quarterly --flat --no-total transfer
2019-10-13 12:39:25.337676849 EEST	hledger-flow End:   hledger --file all-years.journal --period 2017 balance --pretty-tables --quarterly --flat --no-total transfer ExitSuccess (0.693479539s)
2019-10-13 12:39:25.338883089 EEST	hledger-flow Wrote reports/all/2017/transfer-balance.txt
2019-10-13 12:39:25.339109965 EEST	hledger-flow Begin: hledger --file all-years.journal --period 2018 balance --pretty-tables --quarterly --flat --no-total transfer
2019-10-13 12:39:26.022556443 EEST	hledger-flow End:   hledger --file all-years.journal --period 2018 balance --pretty-tables --quarterly --flat --no-total transfer ExitSuccess (0.683417781s)
2019-10-13 12:39:26.023754435 EEST	hledger-flow Wrote reports/all/2018/transfer-balance.txt
2019-10-13 12:39:26.02401414 EEST	hledger-flow Begin: hledger --file all-years.journal --period 2019 balance --pretty-tables --quarterly --flat --no-total transfer
2019-10-13 12:39:26.707729553 EEST	hledger-flow End:   hledger --file all-years.journal --period 2019 balance --pretty-tables --quarterly --flat --no-total transfer ExitSuccess (0.683678604s)
2019-10-13 12:39:26.708550757 EEST	hledger-flow Wrote reports/all/2019/transfer-balance.txt
2019-10-13 12:39:26.708752042 EEST	hledger-flow Begin: hledger --file all-years.journal --period 2020 balance --pretty-tables --quarterly --flat --no-total transfer
2019-10-13 12:39:27.341601174 EEST	hledger-flow End:   hledger --file all-years.journal --period 2020 balance --pretty-tables --quarterly --flat --no-total transfer ExitSuccess (0.63282636s)
Did not write 'reports/all/2020/transfer-balance.txt' (hledger --file all-years.journal --period 2020 balance --pretty-tables --quarterly --flat --no-total transfer) ExitSuccess
2019-10-13 12:39:27.342117833 EEST	hledger-flow Begin: hledger --file all-years.journal --period 2017 incomestatement --depth 2 --pretty-tables not:equity --cost --value
hledger: Flag requires argument: --value (use -h to see usage)
2019-10-13 12:39:27.424352124 EEST	hledger-flow End:   hledger --file all-years.journal --period 2017 incomestatement --depth 2 --pretty-tables not:equity --cost --value ExitFailure 1 (0.082209168s)

=== Begin Error: hledger --file all-years.journal --period 2017 incomestatement --depth 2 --pretty-tables not:equity --cost --value ===
External command:
/home/lestephane/.local/bin/hledger --file /home/lestephane/[...]/all-years.journal --period 2017 incomestatement --depth 2 --pretty-tables not:equity --cost --value
Exit code 1
Error output:
hledger: Flag requires argument: --value (use -h to see usage)

=== End Error: hledger --file all-years.journal --period 2017 incomestatement --depth 2 --pretty-tables not:equity --cost --value ===

The doc says:

--value=TYPE[,COMM]  TYPE is cost, end, now or YYYY-MM-DD.
                          COMM is an optional commodity symbol.
                          Shows amounts converted to:
                          - cost using transaction prices, then optionally to
                          COMM using period-end market prices
                          - period-end market value, in default valuation
                          commodity or COMM
                          - current market value, in default valuation
                          commodity or COMM
                          - market value on the given date, in default
                          valuation commodity or COMM

so TYPE is no longer optional...

This issue is not really blocking me, but I wanted to look at what reports it generates, and start asking questions about how to generate my own (without having to modify the haskell source code obviously).

cannot define --separator option

Is your feature request related to a problem? Please describe.

I would like to import CSV files, which use ; as a separatorchar

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.12.3.0 linux x86_64 ghc 8.6

Is this the latest version?

If your request includes commands you ran and the output, please also include
the runtime options with --show-options e.g:

$ hledger-flow --show-options import
 hledger-flow --show-options import
RuntimeOptions {baseDir = FilePath "/data/", hfVersion = "hledger-flow 0.12.3.0 linux x86_64 ghc 8.6", hledgerInfo = HledgerInfo {hlPath = FilePath "/usr/bin/hledger", hlVersion = "hledger 1.14.2"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [8,6], versionTags = []}}, verbose = False, showOptions = True, sequential = False}

Our Example Statements Repository

https://gist.github.com/weitzj/228ea40f4db33bfe2f49d2d7e43a2871

Can you give examples of what you would like by running hledger-flow on these files?

I would like to have the same behaviour as:

hledger import --separator=";" --file=....

Even better would be that hledger-flow delegates all options to hledger import

Describe alternatives you've considered

Using hledger import --separator=";" and giving up on hledger-flow

3-journal/ files not ending in ".journal" extension are added to yearly include files

Describe the bug
I cheat a bit and add, next to each .journal file in 3-journal, a .png of a related invoice, and a .json of the google text detection api response. These extra files used to be ignored by hledger-flow import, now they get added to the yearly include, causing hledger to fail:

  |
6 | !include 3-journal/2019/2019-05-31-000.png
  |                                           ^
SourcePos {sourceName = "SCRUBBED/Finance/import/personal/wallet/cash/2019-include.journal", sourceLine = Pos 6, sourceColumn = Pos 43} reading SCRUBBED/Finance/import/personal/wallet/cash/3-journal/2019/2019-05-31-000.png:
SCRUBBED/Finance/import/personal/wallet/cash/3-journal/2019/2019-05-31-000.png: hGetContents: invalid argument (invalid byte sequence)

Version and Runtime Information

hledger-flow-v0.14.2.0

Is this the latest version? yes
Please confirm your issue using the latest version of hledger-flow, maybe it has already been fixed.

Also include the runtime options of the command you ran, e.g:

The enable-future-rundir option is now the default, no need to specify it. This option is currently being ignored and will be removed in future.
RuntimeOptions {baseDir = "SCRUBBED/Finance/", importRunDir = "import/personal/wallet/cash/", importStartYear = Nothing, onlyNewFiles = False, hfVersion = "hledger-flow 0.14.2.0 linux x86_64 ghc 8.10", hledgerInfo = HledgerInfo {hlPath = FilePath "SCRUBBED/Finance/hledger", hlVersion = "hledger 1.20.2"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [8,10], versionTags = []}}, verbose = False, showOptions = True, sequential = False}
Collecting input files from SCRUBBED/Finance/import/personal/wallet/cash/
Found 100 input files in 0.024561536s. Proceeding with import...

To Reproduce

Add files in 3-journal as part of a construct script

Expected behavior

One of:

• ignore non- .journal extensions in 3-journal
• make the list of extensions that will be included in the yearly include configurable...

Design the Command Line User Interface

Evaluate all command-line options.
Design a good user interface for both interactive use and scripting.

Does the names of the options make sense? Can we anticipate possible name clashes for future options?

Should some subcommands options be moved to the main options, if they apply to all subcommands?

Import statements in parallel

Right now the input statements are converted into journal files sequentially.

Change the import logic to process them in parallel.

Print the command-line options if requested

Add a command-line switch that will cause the values of all options to be printed.

This will include the basedir being used, and the value of any of the other command-line flags that can be passed in, e.g. the current verbosity.

Unable to run hledger-flow import in a sub-account directory (it always imports everything)

Describe the bug

I used to be able to go to a sub-account directory that had its preprocess construct 1-in etc
and to run hledger-flow import (without specifying an explicit directory), and it would just run the import in that directory. Now as of the latest release, it always imports everything, which takes on my machine 4 seconds for 2017+2018 across all my accounts concurrently. This will add up in time, especially since I'm not done importing all my years and accounts yet, and I use a lot of AWK & PDF conversion utilities in my preprocess and construct scripts. Everything gets re-run everytime.

If I attempt to work around the problem by running hledger-flow import . (that is, explicitly specifying the import directory) from withing the sub-account directory, I get an error (see reproduction below)

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.12.3.0 linux x86_64 ghc 8.6

Is this the latest version? YES

To Reproduce

• Go to the usual sub-account

~/Finance/import/personal/revolut/checking$ ls -1
1-in
2018-include.journal
3-journal
all-years.journal
construct
...

• Check that there are not many files there

~/Finance/import/personal/revolut/checking$ find . -name "*.journal" | wc -l
4

• Run the import command

~/Finance/import/personal/revolut/checking$ hledger-flow import 
Collecting input files...
Found 66 input files in 0.584593834s. Proceeding with import...
Imported 66 journals in 3.849160479s

• It looked for all files starting from ~/Finance/import (Unexpected)

• Attempting to workaround the problem by specifiying the import directory explicitly

~/Finance/import/personal/revolut/checking$ hledger-flow import .
hledger-flow: user error (
Unable to find an hledger-flow import directory at './' (or in any of its parent directories).

Have a look at the documentation for more information:
https://github.com/apauley/hledger-flow#getting-started
)

• It refuses to import (Unexpected)

Expected behavior

~/Finance/import/personal/revolut/checking$ hledger-flow-v0.11.1.2 import
Collecting input files...
Found 2 input files in 0.00324672s. Proceeding with import...
Imported 2 journals in 1.299441531s

A clear and concise description of what you expected to happen.

I'd much rather be able to import a single sub-account at a time. This is especially important when I work on my import scripts, when there is a lot of trial and error and I need a tighter feedback loop.

cabal install error: Not in scope: type constructor 'Rel'

Describe the bug
Using: cabal install hledger-flow
Receive:
Resolving dependencies...
Build profile: -w ghc-8.10.5 -O1
In order, the following will be built (use -v for more details):

• gitrev-1.3.1 (lib) (requires download & build)
• hostname-1.0 (lib:hostname) (requires download & build)
• managed-1.0.8 (lib) (requires download & build)
• mwc-random-0.15.0.1 (lib) (requires download & build)
• optional-args-1.0.2 (lib) (requires download & build)
• path-0.8.0 (lib) (requires download & build)
• profunctors-5.6.2 (lib) (requires download & build)
• semigroupoids-5.3.5 (lib) (requires download & build)
• system-filepath-0.4.14 (lib:system-filepath) (requires download & build)
• path-io-1.6.2 (lib) (requires download & build)
• foldl-1.4.11 (lib) (requires download & build)
• system-fileio-0.3.16.4 (lib) (requires download & build)
• turtle-1.5.22 (lib) (requires download & build)
• hledger-flow-0.14.3.0 (lib) (requires download & build)
• hledger-flow-0.14.3.0 (exe:hledger-flow) (requires download & build)
  Downloading hostname-1.0
  --snipped--
  Completed turtle-1.5.22 (lib)
  Starting hledger-flow-0.14.3.0 (lib)
  Building hledger-flow-0.14.3.0 (lib)

Failed to build hledger-flow-0.14.3.0.
Build log (
/home/reed/.cabal/logs/ghc-8.10.5/hledger-flow-0.14.3.0-fcb2ef806d78720166fe03f1b62892828c13b32c2d2ad0712f200a12cd6af4f9.log
):
Configuring library for hledger-flow-0.14.3.0..
Preprocessing library for hledger-flow-0.14.3.0..
Building library for hledger-flow-0.14.3.0..
[ 1 of 15] Compiling Hledger.Flow.DateTime ( src/Hledger/Flow/DateTime.hs, dist/build/Hledger/Flow/DateTime.o, dist/build/Hledger/Flow/DateTime.dyn_o )
[ 2 of 15] Compiling Hledger.Flow.DocHelpers ( src/Hledger/Flow/DocHelpers.hs, dist/build/Hledger/Flow/DocHelpers.o, dist/build/Hledger/Flow/DocHelpers.dyn_o )
[ 3 of 15] Compiling Hledger.Flow.PathHelpers ( src/Hledger/Flow/PathHelpers.hs, dist/build/Hledger/Flow/PathHelpers.o, dist/build/Hledger/Flow/PathHelpers.dyn_o )
[ 4 of 15] Compiling Hledger.Flow.Import.Types ( src/Hledger/Flow/Import/Types.hs, dist/build/Hledger/Flow/Import/Types.o, dist/build/Hledger/Flow/Import/Types.dyn_o )
[ 5 of 15] Compiling Hledger.Flow.Types ( src/Hledger/Flow/Types.hs, dist/build/Hledger/Flow/Types.o, dist/build/Hledger/Flow/Types.dyn_o )
[ 6 of 15] Compiling Hledger.Flow.Logging ( src/Hledger/Flow/Logging.hs, dist/build/Hledger/Flow/Logging.o, dist/build/Hledger/Flow/Logging.dyn_o )
[ 7 of 15] Compiling Hledger.Flow.BaseDir ( src/Hledger/Flow/BaseDir.hs, dist/build/Hledger/Flow/BaseDir.o, dist/build/Hledger/Flow/BaseDir.dyn_o )
[ 8 of 15] Compiling Hledger.Flow.Common ( src/Hledger/Flow/Common.hs, dist/build/Hledger/Flow/Common.o, dist/build/Hledger/Flow/Common.dyn_o )

src/Hledger/Flow/Common.hs:46:46: error:
• Not in scope: type constructor ‘Rel’
• In the quasi-quotation: [relfile|hledger|]
|
46 | maybeH <- Path.findExecutable [relfile|hledger|]
| ^^^^^^^^^

cabal: Failed to build hledger-flow-0.14.3.0 (which is required by
exe:hledger-flow from hledger-flow-0.14.3.0). See the build log above for
details.
Formating of above is with carats as positioned here, starting at 'h':
Executable [relfile|hledger|]
.............................^^^^^^^^^
Things that work
Can confirm 'hledger' is on path: from same command prompt, 'which hledger' (/home/username/.cabal/bin/hledger), 'hledger --help', 'hledger add' and 'which cabal' (/home/username/.local/bin/cabal) all respond correctly.

Have successfully installed other hledger extensions from same command prompt. For example: 'cabal install hledger-diff' and 'cabal install hledger-iadd' are successfully added.
Expected result
Was expecting the usual: 'Symlinking 'hledger-flow' to …
Install OS and Location
Using Ubuntu v18.04 up to date daily. Command prompt at Gnome terminal. All install is local at '/home/username/' at user $ prompt.

Thank you.

rules lookup should be extended to look for rules in additional paths

Background: I'm defining some rules files for financial institutions that other people may find useful as well (Revolut in EU and soon US, FFB in DE). I would like to share them in GitHub.

Problem: the rules files (.rules, preprocess) must be in the same directory where the yearly sub-directories are located, or in a parent directory. There is no way to 'discover' these files in some other sibling directory that has been fetched from another git repo.

I could use symlinks of course, except that I'm moving to another encryption driver for the statements directory, which does not support symbolic links yet.

So, there could be one external git repo with rules for many institutions, or one repo per institution. And then hledger-flow must know to look into it.

Maybe a HLEDGER_RULE_PATH environment variable?

$ cd GitRepos
$ git clone [email protected]:lestephane/hledger-flow-rules.git
$ tree hledger-flow-rules
hledger-flow-rules
├── ffb
│   └── ffb.rules
└── revolut
    ├── revolut.preprocess
    └── revolut.rules

$ cd ~/Finance
$ vim .envrc
...
path_add  HLEDGER_RULE_PATH  ~/GitRepos/hledger-flow-rules

$ tree Finance
import
├── personal
│   ├── ffb
│   │   └── depot
│   │       ├── 1-in
│   │       │   └── 2018
│   │       │       └── 2018.csv
│   ├── revolut
│   │   ├── checking
│   │   │   ├── 1-in
│   │   │   │   └── 2018
│   │   │   │       └── Revolut-EUR-Statement-2018.csv

import YY as 20YY

Hi, How do I import Y=20 as 2020?
When I use %Y it comes across as 0020

OS X
hledger-flow -v import

contents of rules file:
skip 1

fields date, description, amount
date-format %-m/%-d/%--Y
currency $

error code:
Exit code 1
Error output:
hledger: error: could not parse "6/7/20" as a date using date format "%-m/%-d/%--Y"

Add a logger

Add a new dependency to the project that can log debugging output to log files.

Change the verbose logging to rather use the logger.

This will probably impact #40 - will we then need multiple levels of verbose output to stderr if we have a logger?
Maybe we can add an option to the effect of --log-to-stderr instead of to a file.

Update: RIO has built in logging, consider using this.

Look for hledger on start of execution

Run hledger --version on start of execution to determine if hledger is in the path.

Exit with a helpful error message if hledger cannot be found.

The error message we currently see when running hledger-flow on a system that doesn't have hledger installed isn't very helpful:

hledger: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

Keep the hledger version number around so that it can be used by other parts of the program.
It will be helpful for example to show this version number when printing out the program options (issue #11).

We probably require a specific version (or later) of hledger, although I don't know what the earliest version of hledger is that will work with hledger-flow.

QUESTION: how to break up a transaction/payment?

I have a question about how to use hledger-flow with this situation:

For some of my transactions, I would like to break them down into a group of transactions that are together. For example, if I shop at the hardware store and purchase some tools, and then some supplies for different projects, I'd like to be able to see the high-level transaction for all of those items, the taxes, etc, with the hardware store, but then also be able to see that $X was spent for project1 and $Y was spent for project2 while $Z was spent on tools.

AFAICT, this is possible to do in the PTA journal, but is there a way to do this with hledger-flow's csv imports?

If this is a new feature, I'd be happy to help contribute code to make it work, but would greatly appreciate some guidance / feedback on the best way to extend hledger-flow for this new feature.

Windows: the preprocess and construct scripts are not executed

Describe the bug
When using hledger-flow on windows the preprocess and construct scripts that are called from src\CSVImports.hs (line 70/71) cannot be called.

On Windows executable files or scripts need a file-extension, not just only an executable attribute, hence windows doesn't know what to do with the preprocess and construct scripts.
You could for example try to find a preprocess.bat and construct.bat of preprocess.py and construct.py to get these scripts to execute.

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.13.2.0 mingw32 x86_64 ghc 8.8

This is the last git version I'm running (just updated)

Desktop (please complete the following information):

• OS: Windows10 Pro
• Version 1909