How to Record Incoming Stream of Data

You probebly shall not just delete duplicates, but try to find in the already recorded set where the position of current 300 rows starts.

you could try starting with checking if the bottom 300 in main set is identical with new 300
if not whether bottom 299 is identical with first 299 of 300
if not whether bottom 298 is identical with first 298 of 300
and so on
so in case of sample (8 instead of 300)

main
c
a
b
a
a
b
c
a
b
a
c
b


a
b
a
c
b
a
e
a

start with comparing all new with red (equal count) part in main a and b fits but then is a difference c/a (if there woud be no differences then there was no new data recorded)
so we take 1 less from top in red main and compare with 1 shorter at bottom of new set. it immediately shows not to fit because b is not a
so we take one more less from red and again one shorter from new . again immediately c is not a.
again one less in both (resp top and bottom) and these sets compare exactly the same. we add remaining (not included in comparision) 3 bottom elements of the new data
a
e
a
to the bottom of main set and we are ready for new process