hledger 1.32.3, linux
I have CSV file in UTF-8 format, it starts with BOM <feff>
When I join several such files to one file cat test-bom-*.csv > test-bom.csv, this file contains several BOM characters.
hledger doesn't like those extra BOM characters, it reports an error:
$ hledger -f test-bom.csv print
hledger: error: could not parse "2024-01-02" as a date using date format "%Y-%m-%d"
the CSV record is: "\65279\&2024-01-02", "0.2", "test 2"
the date rule is: %1
the date-format is: %Y-%m-%d
you may need to change your date rule, change your date-format rule, or add a skip rule
for m/d/y or d/m/y dates, use date-format %-m/%-d/%Y or date-format %-d/%-m/%Y
I am not sure but I think that it is not wrong when UTF-8 file has several BOM codes in the file; I tried other utilities and those were not failing with an error. In theory, coding of the file can change in the middle, like from UTF-8 to UTF-16LE...
How to replicate. Prepare test data, several simple CSV files with BOM and without BOM:
for I in 1 2 3; do echo -e "2024-01-0${I},0.${I},test ${I}" > "test-nobom-${I}.csv"; echo -e "\xef\xbb\xbf2024-01-0${I},0.${I},test ${I}" > "test-bom-${I}.csv"; done
cat test-nobom-[123].csv > test-nobom.csv
cat test-bom-[123].csv > test-bom.csv
Files test-bom.csv and test-nobom.csv looks same but they differ in file size:
$ cat test-bom.csv
2024-01-01,0.1,test 1
2024-01-02,0.2,test 2
2024-01-03,0.3,test 3
ls -l test-nobom.csv test-bom.csv
-rw-rw-r-- 1 user user 75 Mar 29 17:59 test-bom.csv
-rw-rw-r-- 1 user user 66 Mar 29 17:59 test-nobom.csv
grep is "confused" with BOM:
$ grep ^2024 test-nobom.csv
2024-01-01,0.1,test 1
2024-01-02,0.2,test 2
2024-01-03,0.3,test 3
$ grep ^2024 test-bom.csv
$ grep ^.2024 test-bom.csv
2024-01-01,0.1,test 1
2024-01-02,0.2,test 2
2024-01-03,0.3,test 3
Create import rules, those are the same, I created test-bom.csv.rules and then used ln -s test-bom.csv.rules test-nobom.csv.rules and ln -s test-bom.csv.rules test-bom-1.csv.rules :
$ cat test-bom.csv.rules
fields date,amount,description
date-format %Y-%m-%d
$ cat test-nobom.csv.rules
fields date,amount,description
date-format %Y-%m-%d
$ cat test-bom-1.csv.rules
fields date,amount,description
date-format %Y-%m-%d
TEST
hledger can import CSV file with single BOM and file without BOM:
$ hledger -f test-bom-1.csv bal
-0.1 income:unknown
0.1 unknown
--------------------
0
$ hledger -f test-nobom.csv bal
-0.6 income:unknown
0.6 unknown
--------------------
0
hledger doesn't like file with several BOM:
$ hledger -f test-bom.csv bal
hledger: error: could not parse "2024-01-02" as a date using date format "%Y-%m-%d"
the CSV record is: "\65279\&2024-01-02", "0.2", "test 2"
the date rule is: %1
the date-format is: %Y-%m-%d
you may need to change your date rule, change your date-format rule, or add a skip rule
for m/d/y or d/m/y dates, use date-format %-m/%-d/%Y or date-format %-d/%-m/%Y