Large External File Usage

Creating desktop or client-server database solutions?

Moderators: heatherlaine, Klaus, FourthWorld, robinmiller, kevinmiller

Mikey
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 579
Joined: Fri Jun 27, 2008 9:00 pm

Re: Large External File Usage

Post by Mikey » Tue Feb 07, 2017 12:01 am

By the way, did you already have the indexes set on the table before you started the import?

AxWald
Posts: 309
Joined: Thu Mar 06, 2014 2:57 pm

Re: Large External File Usage

Post by AxWald » Tue Feb 07, 2017 10:50 am

Well,

this cannot be this complicated? Have you tried it the simple way?
  1. Analyze the data, and determine the data types.
    I see a VARCHAR, a DATETIME, and 12 FLOATs or DOUBLEs.
  2. Create a suitable table. Don't forget an additional ID as col 1:
    SQLite: INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    MySQL: INT(10) NOT NULL AUTO_INCREMENT, set as PRIMARY KEY.
  3. Split a reasonable part of your data for a test.
  4. Fire up your db manager, give it the proper information, and try an import.
  5. Read the error message, correct what you've done wrong.
  6. Once it has imported your test data, empty the table, VACUUM, and give it the full .csv.
If this doesn't work, tell us what exactly went wrong. Then we can see further ;-)

Have fun!

(PS: if you have access to MS Access, this should work too!)
Livecode programming until the cat hits the fan ...

MaxV
Posts: 1487
Joined: Tue May 28, 2013 2:20 pm
Location: Italy
Contact:

Re: Large External File Usage

Post by MaxV » Thu Feb 09, 2017 1:11 pm

Not a lot of thought wrote:I have a large file 10,000,000+ records roughly 3.5 Gigs. It is in CSV format....
I suggest you to use a:

Code: Select all

BEGIN TRANSACTION;
all you MySQL comamnds;
...
COMMIT;
It would solve your problem.
However a repeat for each line tLine in CSV would resolve too.
Livecode Wiki: http://livecode.wikia.com
My blog: http://livecodeitalia.blogspot.it
To post code use this: http://tinyurl.com/ogp6d5w

Post Reply

Return to “Databases”