How to read the last line in a large textfile

[KalleKL]

Hello all,

I have a large textfile (up to 20 000 000 rows), how can I read the last line fastest?

KalleK

[cytop]

The suggestion in your other thread (ADO) is probably the easiest, but if the file contains fixed length records (All lines exactly the same size) then there is another possibility... Is it?

[KalleKL]

Hello cytop, no it is not fixed length. It is csv file with number and date and time. How do I use ADO to read?

[cytop]

The following should return the details from the last record in a CSV/TXT file. Minor changes are needed if another delimiter is used and it assumes the first line contains headings (although that won't really matter as you only want the last line.

It requires a reference to Microsoft ActiveX Data Objects (Tested with 6.1)

Please Login or Register  to view this content.

[KalleKL]

Thanks! I will test and compare the times. Is is posibble with ADO to start to read from a line? Say line 40000 of a file with 2 000 000 lines

[cytop]

You can use the MOVE method, or one of it's variants.

Please Login or Register  to view this content.

In all cases I am ignoring the fact you might try to move before the first or after the last record - this will raise an error so you need to include error handling.

IF you only want to pick particular records - for example where a 'NAME' column contains 'Smith', you can specify that in the initial selection:

Please Login or Register  to view this content.

You can then move between those records matching 'SMITH' but it's beginning to get a little complicated now.

[KalleKL]

Thanks cytop! I get it to work and see if it is faster then my curret solution to find the end of the file. Thanks again for the help!

[cytop]

There is another method where you open the file in Binary mode, skip to the last position and then skip back 1 character at at time until you hit a Carriage Return or Line Feed character.

For example

Please Login or Register  to view this content.

This is the basis for what I mentioned earlier about fixed record lengths. If each line was exactly the same length it would be possible to calculate the offset position of any physical line directly. As is, it can really only be used to return the last line. Or perhaps the last 2nd/3rd and so on, lines but it begins to get messy.

The main advantage of this method is that positioning at the end of the file is instantaneous and then it only has to loop backwards to find the terminator for the previous line.

There may be an issue where the last line does not terminate with a Carriage Return or Line Feed. This is something you are going to have to check yourself and make any necessary changes to the code that sets the initial position of the pointer at the start of the loop.