Needing to run songbook filler exe on linux, any good walkthrus? :)

[Hoppa_Joel]

Needing to run songbook filler exe on linux, any good walkthrus?
I dont know how to use lutris, so if that is the answer please give me easy to use newbie instructions hehe

[Thurallor]

Needing to run songbook filler exe on linux, any good walkthrus?
I dont know how to use lutris, so if that is the answer please give me easy to use newbie instructions hehe

Try this.

If it doesn't work, let me know of any error messages. If I thought someone was going to use the script, I might be persuaded to expend the effort to update it.

[BRaymarK]

IDK how married you are to the idea of using the Songbook filler. At one point I got frustrated with issues I had with other tools, so I wrote a perl script. I had trouble with a number of issues with .abc files and so had the script tell me about them so they could be fixed. I would not say it is overly fancy or anything, but it was good enough for me to get a lot of songs in my songbook. Let me know if you are interested. I've never made plugins before, so its just a standalone script. Help:

  Usage:
    ${Help_Name} [ Options ] AccountName [ Server ]

    For optimal results, install (and run) this script from somewhere under
    the user's game data folder.  A good location is the Music directory or
    the Songbook plugin folder under a user's "The Lord of the Rings Online"
    folder usually found in the user's "Documents" folder.  As long as there
    is a "The Lord of the Rings Online" folder in the path where the script
    is installed, the script should be able to automatically determine where
    to put the song file.  It is not recommended to put the script under the
    folder tree where the LOTRO client is installed, but if the script is put
    somewhere else, use --bin="/path/to/songbook-plugin" option to help the
    script determine where user's LOTRO client configuration files are stored.

    "AccountName" is the login name used when launching the game client.  The
    name is required in order to place the generated file in an appropriate
    sub-folder of the PluginData directory.  The name is case-sensitive on
    non-Windows systems.

    "Server" is supplied only when it is important to generate the song file
    only for a particular server.  The default is to install the generated
    file in the "AllServers" PluginData folder so it is available no matter
    which game server is used.

    If the LOTRO client 'Music', 'Plugins', and 'PluginData' folder are not
    automatically found, run the script from the Songbook plugin folder, or,
    use --dir= to identify the directory where they reside.

    While the script is running, if (known) anomalies are found in .abc files,
    they are reported with file name and line number where the condition was
    found.  A few anomalies (per the ABC music file standard) are known not to
    negatively affect the LOTRO client's ability to play music.  In this case,
    messages indicate the issue simply to raise awareness of the fact that the
    file is slightly out-of-specification.

    Though not generally recommended, custom paths may be set as follows:

      --bin=
        (Try to) set sensible defaults for all paths, presuming that the path
        contains a directory named "The Lord of the Rings Online" that is the
        user's LOTRO client configuration data folder.
      --lib=
        Set the location for Music files from which to make the songbook.
      --out=
        Set the location where the songbook file should be written.
  Options:
    [ --dir= ]
      Specify where the LOTRO client 'Music', 'Plugins', and 'PluginData'
      folders are located.
    [ --bin= ]
    [ --lib= ]
    [ --tmp= ]
    [ --out= ]
      Various working directories that all default to a path used to invoke
      this script.  --tmp= and --out= determine locations for intermediate and
      result files respectively, while --lib= usually specified where input
      files are expected.  --bin often is used to change multiple directory
      settings so set it first when multiple directory switches are used.
        Current bin: '$Bin_Dir'
        Current lib: '$Lib_Dir'
        Current out: '$Out_Dir'
        Current tmp: '$Tmp_Dir'
    [ --debug= ]
      Enable diagnostic logging and/or script tracing during development.
      Optionally assign an integer debug level (default 1, max $DebugMax).
      Output detail and/or quantity vary by numeric level (if supported).
        Current level: '$Debug'
    [ --stderr= ]
      Direct logged diagnostic/error information to an alternate destinaion.
        Default path: '/dev/stderr'
        Current path: '$StdErr'
    [ --version ]
      Display version and exit after parsing command-line switches.
    [ --help ]
      Display this information and exit after parsing command-line switches.

[Hoppa_Joel]

Thank you both,
I am giving a try to both methods, however I am not sure how to launch the files.
I created them as .pl but this just opens up as a notepad file and doesn't launch. ( I have set the file to executable )

[Thurallor]

You will need to open the terminal and enter the command (see instructions).

[Hoppa_Joel]

Oops, dolphin doesnt like running things in terminal, had to use nemo to broswe to location, but i am getting errors that pop up and close terminal before I can read them.

[Thurallor]

So in terminal, I'm running this:

/home/hoppa/.wine/drive_c/users/hoppa/My Documents/The Lord of the Rings Online/Plugins/Chiran/discover_songs.pl
which is the copied the path, then typed in the file name.
but I get this:
bash: /home/hoppa/.wine/drive_c/users/hoppa/My: No such file or directory

So Hmm.. just not understanding how to launch this.

Do it like this:

cd "/home/hoppa/.wine/drive_c/users/hoppa/My Documents/The Lord of the Rings Online/Plugins/Chiran"

and then

perl ./discover_songs.pl

[Hoppa_Joel]

Here is what I get :
syntax error at ./discover_songs.pl line 111, at EOF
Execution of ./discover_songs.pl aborted due to compilation errors.

[Hoppa_Joel]

ahh I think I got it, there was a . after pl at the end.
I ran it again,

Selected my music directory, and got this:

-> /home/hoppa/.wine/drive_c/users/hoppa/My Documents/The Lord of the Rings Online/PluginData/********/AllServers/SongbookData.plugindata
cp: target 'Online/PluginData/*********/AllServers/' is not a directory

******** = my account.

I do have allservers folder

[Thurallor]

ahh I think I got it, there was a . after pl at the end.
I ran it again,

Selected my music directory, and got this:

-> /home/hoppa/.wine/drive_c/users/hoppa/My Documents/The Lord of the Rings Online/PluginData/********/AllServers/SongbookData.plugindata
cp: target 'Online/PluginData/*********/AllServers/' is not a directory

******** = my account.

I do have allservers folder

Great, the script ran all the way to the end, but just a minor bug prevented it from copying the file into the correct folder.

I uploaded a new version of the script (same place) which should fix this problem.

[Hoppa_Joel]

very cool, I downloaded and it looks like it processed! I will launch lotro and have a look about in songbook, thank you!

[Hoppa_Joel]

So, after running things, ( which did build files ) Songbook didnt load, and I get this message:

...the Rings Online/PluginData/*********/AllServers/SongbookData.plugindata:147: '<eof>' expected near 'Name'
...gs Online\Plugins\Chiran\Songbook \SongbookWindow.lua:43: Unable to parse file!
Unable to load "songbook".

I think the version of songbook I am using is the latest ( updated last year, version 93 ) the one by Chiran. I haven't tried another version so maybe I am using the wrong one? Let me know!

[BRaymarK]

So, after running things, ( which did build files ) Songbook didnt load, and I get this message:

...the Rings Online/PluginData/*********/AllServers/SongbookData.plugindata:147: '<eof>' expected near 'Name'
...gs Online\Plugins\Chiran\Songbook \SongbookWindow.lua:43: Unable to parse file!
Unable to load "songbook".

I think the version of songbook I am using is the latest ( updated last year, version 93 ) the one by Chiran. I haven't tried another version so maybe I am using the wrong one? Let me know!

Unable to parse file is a clue that the file is being written incorrectly based on anomalies in some .abc files. Among the problems that the tools didn't manage:

1) Byte-Order-Mark characters in song files messed things up.
2) Some .abc files erroneously use x: instead of X: (LOTRO seems to not take issue with lowercase x
3) Files with multiple X: had to be treated differently.
4) Some .abc files didn't have the (X.

In most cases, LOTRO was more forgiving of the anomalies in the .abc files than the songbook maker was, and the issues that tripped up the songbook maker actually did create gnarly syntax issues.

[Thurallor]

Unable to parse file is a clue that the file is being written incorrectly based on anomalies in some .abc files. Among the problems that the tools didn't manage:

1) Byte-Order-Mark characters in song files messed things up.
2) Some .abc files erroneously use x: instead of X: (LOTRO seems to not take issue with lowercase x
3) Files with multiple X: had to be treated differently.
4) Some .abc files didn't have the (X.

In most cases, LOTRO was more forgiving of the anomalies in the .abc files than the songbook maker was, and the issues that tripped up the songbook maker actually did create gnarly syntax issues.

Thanks. I have modified the script; the new version is in the same place.

It will now handle "x:" instead of "X:". It already should handle files with multiple "X:"s. Files with the BOM at the beginning should work, as long as there are comments at the top; those are skipped anyway.

Any other problems aren't handled, but at least now you will get an error message telling you which file it has a problem with. You can then remove that file from the Music directory and run the script again.

[Hoppa_Joel]

Ran it multiple times ( removing old files )
til it finally compiled.
went into lotro and could not load Songbook ( got this error )
...gs Online\Plugins\Chiran\Songbook \SongbookWindow.lua:1098: attempt to index field '?' (a nil value)


It does seem we are getting there, a step at a time

Thank you both for helping get this resolved

[Thurallor]

Ran it multiple times ( removing old files )
til it finally compiled.
went into lotro and could not load Songbook ( got this error )
...gs Online\Plugins\Chiran\Songbook \SongbookWindow.lua:1098: attempt to index field '?' (a nil value)


It does seem we are getting there, a step at a time

Thank you both for helping get this resolved

Would you mind sending me your SongbookData.plugindata file (there should be a copy in Plugins/Chiran)? It would expedite my debugging. (I sent you my e-mail address in a PM.)

[Hoppa_Joel]

So, when I got up this morning. I had a thought "what if the file was just corrupted"
So instead ot logging into my main account, I logged into a free to play one, that I occasionally do music with.

Everything was fine, the song changes, the new song I added etc.
I went back to main account " could not load" So, I delete the song data file, run the compiler again that you built, and BOOM! it works

Then I logged in here, and see you wanted the faulty file... ooops! I deleted that one.


It looks like everything is working now. Thank you very much

[BRaymarK]

Thanks. I have modified the script; the new version is in the same place.

It will now handle "x:" instead of "X:". It already should handle files with multiple "X:"s. Files with the BOM at the beginning should work, as long as there are comments at the top; those are skipped anyway.

Any other problems aren't handled, but at least now you will get an error message telling you which file it has a problem with. You can then remove that file from the Music directory and run the script again.

So, oddly, your script says to delete an invalid song file that my script doesn't mind at all. It's not necessary to delete these files. LOTRO can play them. Ah... yeah, your script cannot handle BOM characters:

[/home/apps/LOTRO/Plugins/Chiran]$ od -a -tx1 ../../Music/Instruments/Feathers2Bagpipes.abc | head -1
0000000   o   ;   ?   X   :   1  cr  nl   T   :   T   o   s   s  sp   t
         ef  bb  bf  58  3a  31  0d  0a  54  3a  54  6f  73  73  20  74

You need to ignore the 0xef, 0xbb, 0xbf at the beginning of a song file.

    while (<$Abc>)
    {
      chomp $_;
      $AbcLine = $_;
      $LineNmbr += 1;
      #
      # https://abcnotation.com/wiki/abc:standard:v2.1#abc_file_identification
      #
      # "Note for developers: Software should ignore the byte order mark (BOM)
      # if encountered as the first character of the file."
      #
      if ($LineNmbr == 1)
      {
        $AbcLine =~ s/^\x{EF}\x{BB}\x{BF}//g;
      }

-or- the mod I made to yours:

    foreach $line (<$abcfile>) {
        $line =~ s/^\x{EF}\x{BB}\x{BF}//g;

Now I get further with discover_songs.pl, but still get stuck on another .abc file that begins:

T: Andalusian Air
K: C
L: 1/8
Q:1/4=160

So I guess it can't handle files without X: even though LOTRO is okay with that.

Here's the heart of my .abc parser:

# BEGIN Tracks ---------------------------------------------------------------

    open($Abc, "<", "${Lib_Dir}/${SongPath}${SongFile}.abc")
      or die "\nOOPS:  Cannot open '${Lib_Dir}/${SongFile}.abc'";

    undef $TrackName;
    undef $TrackId;
    $TrackNmbr = 0;
    $LineNmbr = 0;
    while (<$Abc>)
    {
      chomp $_;
      $AbcLine = $_;
      $LineNmbr += 1;
      #
      # https://abcnotation.com/wiki/abc:standard:v2.1#abc_file_identification
      #
      # "Note for developers: Software should ignore the byte order mark (BOM)
      # if encountered as the first character of the file."
      #
      if ($LineNmbr == 1)
      {
        $AbcLine =~ s/^\x{EF}\x{BB}\x{BF}//g;
      }
      #
      # X: is the field for Id, but x: was observed "in the wild".  Per the
      # standard above, the field name is case-sensitive.  If x:Id is seen,
      # give a hint as to why an ID is not detected in a .abc file.
      #
      if ($AbcLine =~ m/^x:\s*(\d+)/)
      {
        ErrorLog
          "${SongPath}${SongFile}.abc: Line $LineNmbr: Invalid Id (x: vs X:)";
      }
      # Even though only X: is valid per the .abc file standard, the LOTRO
      # client appears not to care if x: is in the file.
      #
      if ($AbcLine =~ m/^X:\s*(\d+)/)
      {
        if (defined($TrackId) and defined($TrackName))
        {
          # There is more than one X: in the file, so write out a track and
          # set the Id for the next track.
          #
          $TrackNmbr += 1;
          $SongbookBuffer .= TrackInfo $TrackNmbr, $TrackId, $TrackName;
          undef $TrackName;
          undef $TrackId;
        }
        if (not defined($TrackName))
        {
          # The first (or only) track in a file
          #
          $TrackId = $1;
        }
        else
        {
          ErrorLog
            "${SongPath}${SongFile}.abc: Line $LineNmbr: Missing Name?";
        }
      }
      elsif ($AbcLine =~ m/^T:\s*([[:print:]]+)/)
      {
        # The .abc file standard doesn't allow for X: to be missing (though it
        # does allow X: to not have a value).  Even so, the LOTRO client does
        # not appear to actually care if X: is missing (though it might have
        # issues if a file contains more than one track).
        #
        if ( not defined($TrackId) and $TrackNmbr == 0 )
        {
          ErrorLog
            "${SongPath}${SongFile}.abc: Line $LineNmbr: no track ID: " .
              "using X:1";
          $TrackId = "1";
        }
        if ( defined($TrackId) )
        {
          $TrackName .= " " if length($TrackName);
          $TrackName .= $1;
          $TrackName =~ s/"/\\"/g;
        }
        else
        {
          ErrorLog "${SongPath}${SongFile}.abc: Line $LineNmbr: Missing Id?";
        }
      }
    }
    if (defined($TrackId) and defined($TrackName) )
    {
      $TrackNmbr += 1;
      $SongbookBuffer .= TrackInfo $TrackNmbr, $TrackId, $TrackName;
      undef $TrackName;
      undef $TrackId;
    }

    close($Abc);

# END   Tracks ---------------------------------------------------------------

[Thurallor]

You need to ignore the 0xef, 0xbb, 0xbf at the beginning of a song file.

Done, thanks.

Why don't you upload yours to Google Drive and post a link here? Maybe someone who finds this thread in the future will be able to make your script work in cases where mine won't.

[Hoppa_Joel]

oh is there a new version for me to download?
The last one did work
thank you both again.

[BRaymarK]

Since I took a different approach... i.e. putting songs into a buffer before writing them to a file, I guess is easier for my script to substitute missing X:

To fix yours with missing X: when a song file has only one song, but I'm guessing this will miss others:

    foreach $line (<$abcfile>) {
       $outline="";
       if ($line =~ m/^[\xef\xbb\xbf]*X:\s*(\d+)/i) {
            $id = $1;
            print $outfile "\t\t\t\t{\n";
            print $outfile "\t\t\t\t\tId = \"${id}\";\n";
            $tracknum++;
            $x++;
        } elsif ($line =~ m/^T: *([[:print:]]+)/i) {
             if ($tracknum == 1 and $x == 0 and $t == 0) {
              $id = 1;
              print $outfile "\t\t\t\t{\n";
              print $outfile "\t\t\t\t\tId = \"1\";\n";
              $tracknum++;
              $x++;
            }
            $name = $1;
            $name =~ s/"/\\"/g;
            print $outfile "\t\t\t\t\tName = \"${name}\";\n";
            print $outfile "\t\t\t\t};\n";
            $t++;
        }
    }
    close($abcfile);

Yes, it doesn't fix everything...

Malformed ABC file: ./Music/Raymionds/butterboys.abc
Please remove this file and run the script again.

The issue your script has with that file is that there are multiple T: records for one X: record. What my script does is merge T: record content together when more than one occur for one X: record.
A simple fix for that might be to just ignore extra T: records along the lines of:

    foreach $line (<$abcfile>) {
       if ($line =~ m/^[\xef\xbb\xbf]*X:\s*(\d+)/i) {
            $id = $1;
            print $outfile "\t\t\t\t{\n";
            print $outfile "\t\t\t\t\tId = \"${id}\";\n";
            $tracknum++;
            $x++;
        } elsif ($line =~ m/^T: *([[:print:]]+)/i) {
             if ($tracknum == 1 and $x == 0 and $t == 0) {
              $id = 1;
              print $outfile "\t\t\t\t{\n";
              print $outfile "\t\t\t\t\tId = \"1\";\n";
              $tracknum++;
              $x++;
            }
            next if $x == $t;
            $name = $1;
            $name =~ s/"/\\"/g;
            print $outfile "\t\t\t\t\tName = \"${name}\";\n";
            print $outfile "\t\t\t\t};\n";
            $t++;
        }
    }
    close($abcfile);

Yes, this gets further... but now:

Malformed ABC file: ./Raymionds/CarrigHarp.abc
Please remove this file and run the script again.

This file is <CR> delimited instead of <CR><LF> delimited. Not sure what to suggest there without thinking on it a bit more.

[BRaymarK]

oh is there a new version for me to download?
The last one did work
thank you both again.

The new version would let you keep more .abc files instead of deleting them.

[Thurallor]

Well, if my script works for well-formed ABC files, which is probably more than 95% of them, I'm ok with it. The files are easy enough to correct once you know the short list of errors that are acceptable to LOTRO. However, if you want to modify it, you can distribute it, or I am happy to upload a new version to that link. I just don't have any more time right now to spend on testing it. (But I don't know why you'd rather do that than just upload your own script, which already can handle malformed ABC files!)

[BRaymarK]

I don't know why any .abc files should be removed if LOTRO doesn't have a problem with them, but it surely is up to you what you do with your script.

Almost 6000 files in my collection read with no error (and no deletions required) after making this mod:

    print $outfile "\t\t{\n";
    print $outfile "\t\t\tFilepath = \"${dirs}\";\n";
    print $outfile "\t\t\tFilename = \"${filename}\";\n";
    print $outfile "\t\t\tTracks = {\n";
    $delim = $\;
    open F, $filepath or die "Can't read $filepath: $!";
    undef $/;
    $filedata = <F>;
    close F;
    $delim = $\;
    @linedata = split /\R/, $filedata;
    $tracknum = 1;
    $x = 0; $t = 0;
    foreach $line ( @linedata ) {
       if ($line =~ m/^[\xef\xbb\xbf]*X:\s*(\d+)/i) {
            $id = $1;
            print $outfile "\t\t\t\t{\n";
            print $outfile "\t\t\t\t\tId = \"${id}\";\n";
            $tracknum++;
            $x++;
        } elsif ($line =~ m/^T: *([[:print:]]+)/i) {
            if ($tracknum == 1 and $x == 0 and $t == 0) {
              $id = 1;
              print $outfile "\t\t\t\t{\n";
              print $outfile "\t\t\t\t\tId = \"1\";\n";
              $tracknum++;
              $x++;
            }
            next if $x == $t;
            $name = $1;
            $name =~ s/"/\\"/g;
            print $outfile "\t\t\t\t\tName = \"${name}\";\n";
            print $outfile "\t\t\t\t};\n";
            $t++;
        }
    }
    print $outfile "\t\t\t}\n";
    print $outfile "\t\t};\n";

I PM'd you the whole modded file.

[Hoppa_Joel]

So what I notice is that it compiled,
it works with One character, if I log in another character, on same account. ( when loading the songbook )
I get:
...gs Online\Plugins\Chiran\Songbook \SongbookWindow.lua:1098: attempt to index field '?' (a nil value)

I don't understand why this is happening, as the song data should not effect the plugin layout, but it is corrupting the layout deleting the layout fixes it temporarily then it is corrupted for next character, ( still works for the first one )