CSV URL Import Creating Duplicate Records


#1

We use the CsvUrlImport to fetch monthly billing data.

Ignore Duplicate Timestamps, Ignore Duplicate Records are all set to TRUE.

COV is set to TRUE.

If we run the poller consecutive times with the same import data, the very last record is duplicated for each time it runs.

This may be a bug in the Niagara history database, but I would expect the above setting would prevent this.

Interestingly, if we attempt to edit the history and set the duplicate records to zero, the editor only allows us to update the last record.

Thoughts?


#2

I assume the duplication is the same value with different timestamps based on that description? If that is the case, can you confirm the timestamp configuration in this import is using the timestamp column from the imported data and not the timestamp of the time of the import?

History editing in Niagara is has fairly limited functionality wouldn’t recommend this as the solution for this issue.


#3

It’s the same value and same timestamp. The timestamp is base on the imported data and is typically in the past, so it’s definitely not using the time of import.


#4

Can you provide a screen shot of one of the history extensions experiencing this issue so I can review all the settings?


#5

Thanks for your help.

CsvUrlImport


#6

Are you able to provide a sample output from the URL and also the current history where the duplicates are happening?


#7

Here: https://app.box.com/s/6cd0wsrtdlt8l8bvyjc59hsvyrygv2o4

CsvUrlImportHistory.csv: Niagara history with last record duplicated
CsvUrlImportSource.csv: Source data served by URL


#8

If you were to manually do another poll, does this add a third duplicate entry or is does this only happen with the one duplicate?


#9

It will continue to add another duplicate record (both timestamp & value) after each subsequent poll (both manual and triggered).


#10

I have been able to successfully reproduce this issue with the files provided. I will update as soon as I have a solution.


#11

Good to hear, Jonathan. Thanks for the help.


#12

I believe I have tracked down the issue and it relates to the way the last record in the history is being resolved. I will be posting an update shortly.

There is also a work around in the current version. If you can guarantee the file contents from that URL contain the entire historical record as it appears to, you could turn on the clearHistoryWhenOlder option. This would rebuild the entire history in Niagara on every subsequent poll with the full contents of that csv file.


#13

You can download the updated version (4.2.36.34.5) from here